2024-03-25T18:20:59.276 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T18:20:59.277 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_amathuri@teuthology --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/orig.config.yaml 2024-03-25T18:20:59.319 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:20:59.400 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621322' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi130.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC151G++LGPgESxw1yLJIi18NohTBabOkdI3F3RowAdjaQ+fN969rAq2YN8RW/M5KRLIbbjm23W8VU5H6WnoUZA= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoqCjQcNIhRg3dFSzdt0NrGct9OOSowWbTVZL//H+1UrhCpY21F4j7tYptMsoDnwA2eSnEepdYgb6Ac2LGNpcQ= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T18:20:59.401 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T18:20:59.401 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T18:20:59.401 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T18:20:59.403 INFO:teuthology.task.internal:Checking packages... 2024-03-25T18:20:59.426 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T18:20:59.426 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T18:20:59.426 INFO:teuthology.packaging:ref: None 2024-03-25T18:20:59.426 INFO:teuthology.packaging:tag: None 2024-03-25T18:20:59.426 INFO:teuthology.packaging:branch: main 2024-03-25T18:20:59.426 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:20:59.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T18:20:59.805 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T18:20:59.807 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T18:20:59.814 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T18:20:59.814 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T18:20:59.847 INFO:teuthology.task.internal:Saving configuration 2024-03-25T18:20:59.862 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T18:20:59.869 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:20:59.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:15:20.298507', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC151G++LGPgESxw1yLJIi18NohTBabOkdI3F3RowAdjaQ+fN969rAq2YN8RW/M5KRLIbbjm23W8VU5H6WnoUZA='} 2024-03-25T18:20:59.916 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:15:20.299504', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoqCjQcNIhRg3dFSzdt0NrGct9OOSowWbTVZL//H+1UrhCpY21F4j7tYptMsoDnwA2eSnEepdYgb6Ac2LGNpcQ='} 2024-03-25T18:20:59.917 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T18:20:59.924 INFO:teuthology.task.internal:roles: ubuntu@smithi130.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T18:20:59.924 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T18:20:59.924 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T18:20:59.999 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc3ea3f60d0>, signals=[15]) 2024-03-25T18:20:59.999 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T18:21:00.006 INFO:teuthology.task.internal:Opening connections... 2024-03-25T18:21:00.007 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T18:21:00.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:21:00.106 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T18:21:00.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:21:00.196 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T18:21:00.208 DEBUG:teuthology.orchestra.run.smithi130:> uname -m 2024-03-25T18:21:00.225 INFO:teuthology.orchestra.run.smithi130.stdout:x86_64 2024-03-25T18:21:00.225 DEBUG:teuthology.orchestra.run.smithi130:> cat /etc/os-release 2024-03-25T18:21:00.281 INFO:teuthology.orchestra.run.smithi130.stdout:NAME="CentOS Stream" 2024-03-25T18:21:00.281 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION="9" 2024-03-25T18:21:00.281 INFO:teuthology.orchestra.run.smithi130.stdout:ID="centos" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:VERSION_ID="9" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:ANSI_COLOR="0;31" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:21:00.282 INFO:teuthology.orchestra.run.smithi130.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:21:00.283 INFO:teuthology.lock.ops:Updating smithi130.front.sepia.ceph.com on lock server 2024-03-25T18:21:00.309 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-03-25T18:21:00.327 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-03-25T18:21:00.327 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-03-25T18:21:00.383 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T18:21:00.384 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T18:21:00.384 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-03-25T18:21:00.409 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T18:21:00.422 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T18:21:00.451 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T18:21:00.451 DEBUG:teuthology.orchestra.run.smithi130:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:21:00.453 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-03-25T18:21:00.469 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T18:21:00.480 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T18:21:00.480 DEBUG:teuthology.orchestra.run.smithi130:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:21:00.510 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-03-25T18:21:00.582 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T18:21:00.638 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T18:21:00.656 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T18:21:00.657 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T18:21:00.657 DEBUG:teuthology.orchestra.run.smithi130:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:21:00.657 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T18:21:00.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:00.674 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-25T18:21:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:21:00.675 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-25T18:21:00.730 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:21:00.731 INFO:teuthology.task.kernel:Running kernel on smithi130: 5.14.0-361.el9.x86_64 2024-03-25T18:21:00.731 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum install -y kernel 2024-03-25T18:21:00.732 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-361.el9.x86_64 2024-03-25T18:21:00.732 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.14.0-361.el9.x86_64 2024-03-25T18:21:00.732 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum install -y kernel 2024-03-25T18:21:02.779 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:21:02.892 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T18:21:04.196 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-25T18:21:04.267 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-25T18:21:07.483 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 2.7 MB/s | 8.0 MB 00:02 2024-03-25T18:21:08.329 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-03-25T18:21:12.569 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 4.7 MB/s | 19 MB 00:04 2024-03-25T18:21:13.458 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 19 MB 00:04 2024-03-25T18:21:19.681 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 1.7 MB/s | 6.1 MB 00:03 2024-03-25T18:21:20.743 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 1.6 MB/s | 6.1 MB 00:03 2024-03-25T18:21:21.127 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T18:21:22.247 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 45 kB/s | 16 kB 00:00 2024-03-25T18:21:23.449 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T18:21:24.086 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 21 MB 00:02 2024-03-25T18:21:28.366 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-25T18:21:28.903 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T18:21:30.514 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:21:30.514 INFO:teuthology.orchestra.run.smithi163.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:21:30.611 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:21:30.621 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:21:30.622 INFO:teuthology.orchestra.run.smithi163.stdout:Install 4 Packages 2024-03-25T18:21:30.622 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:21:30.622 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 76 M 2024-03-25T18:21:30.622 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 123 M 2024-03-25T18:21:30.622 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:21:30.725 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.725 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.726 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.867 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.879 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.886 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:30.921 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:30.921 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:30.956 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:30.957 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:31.001 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T18:21:31.001 INFO:teuthology.orchestra.run.smithi130.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T18:21:31.002 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:31.008 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:31.016 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:31.033 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:31.036 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:31.072 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.097 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.101 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.102 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-03-25T18:21:31.108 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:Install 4 Packages 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 76 M 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 123 M 2024-03-25T18:21:31.109 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:21:31.211 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.220 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.245 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.373 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.379 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.406 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.534 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.535 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.536 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.725 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.749 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.787 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.864 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:31.881 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.892 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:31.904 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.927 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:31.928 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:31.939 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:31.947 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.968 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:31.975 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.012 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.019 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.026 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:32.048 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.057 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.065 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.083 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:32.091 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:32.101 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.108 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.118 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.135 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.167 INFO:teuthology.orchestra.run.smithi163.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 44 kB/s | 66 kB 00:01 2024-03-25T18:21:32.170 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.179 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.193 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:32.210 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.260 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.278 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.357 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.369 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.424 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.444 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.481 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.497 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:32.530 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:32.559 INFO:teuthology.orchestra.run.smithi163.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 7.9 MB/s | 15 MB 00:01 2024-03-25T18:21:32.564 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.565 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.582 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.583 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.583 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:32.597 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:32.615 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.650 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.653 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.667 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.686 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:32.733 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:32.754 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.757 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.767 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:32.843 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.844 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.844 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.854 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:32.963 INFO:teuthology.orchestra.run.smithi130.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 42 kB/s | 66 kB 00:01 2024-03-25T18:21:32.963 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:32.997 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:33.395 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:33.409 INFO:teuthology.orchestra.run.smithi163.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T18:21:33.465 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:33.546 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:33.602 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:33.605 INFO:teuthology.orchestra.run.smithi130.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 6.7 MB/s | 15 MB 00:02 2024-03-25T18:21:33.723 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:21:33.924 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:34.042 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:21:34.094 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:34.164 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:34.227 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:34.475 INFO:teuthology.orchestra.run.smithi163.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 12 MB/s | 27 MB 00:02 2024-03-25T18:21:34.480 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:34.480 INFO:teuthology.orchestra.run.smithi163.stdout:Total 20 MB/s | 76 MB 00:03 2024-03-25T18:21:34.530 INFO:teuthology.orchestra.run.smithi130.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 11 MB/s | 33 MB 00:03 2024-03-25T18:21:34.563 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:21:34.617 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:21:34.636 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:21:34.776 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] kernel-modules-core-5.14.0-432.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-core-5.14.0-432.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:21:35.140 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:21:35.480 INFO:teuthology.orchestra.run.smithi130.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 11 MB/s | 27 MB 00:02 2024-03-25T18:21:35.485 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:35.485 INFO:teuthology.orchestra.run.smithi130.stdout:Total 17 MB/s | 76 MB 00:04 2024-03-25T18:21:35.529 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:21:35.529 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:21:36.145 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:21:36.525 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:21:36.525 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:21:36.543 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:21:36.543 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:21:37.538 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:21:37.539 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:21:39.000 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:21:39.629 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:21:39.711 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:21:40.025 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:21:40.732 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:21:40.815 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:21:42.434 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:21:42.562 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:21:43.557 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:21:43.695 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:21:47.235 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:21:47.295 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:21:48.378 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:21:48.478 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:21:52.040 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:21:53.233 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:21.232 INFO:teuthology.orchestra.run.smithi163.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:22:21.232 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:22:21.276 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:21.635 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:21.635 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:21.635 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:21.635 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:22.077 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:22.077 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:22:22.077 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:22:22.077 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.077 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.078 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.078 INFO:teuthology.orchestra.run.smithi163.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.078 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:22:22.078 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:22:22.430 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:22.430 INFO:teuthology.orchestra.run.smithi130.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T18:22:22.430 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:22:22.433 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:22:22.475 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:22.544 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.545 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:22:22.545 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:22:22.545 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:22:22.545 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:22:22.545 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T18:22:22.546 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi163.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:22:22.546 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:22:22.625 INFO:teuthology.orchestra.run.smithi163.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:22.626 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:22:22.626 DEBUG:teuthology.orchestra.run.smithi163:> sudo rpm -qi grub2-tools 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Name : grub2-tools 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Epoch : 1 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Version : 2.06 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Release : 68.el9 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: x86_64 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Group : Unspecified 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Size : 8264254 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:License : GPLv3+ 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:22:22.666 INFO:teuthology.orchestra.run.smithi163.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:Packager : builder@centos.org 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:Vendor : CentOS 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:Summary : Support tools for GRUB. 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:Description : 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:hardware devices. 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:22:22.667 INFO:teuthology.orchestra.run.smithi163.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:22:22.668 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:22:22.668 INFO:teuthology.task.kernel:Updating grub on smithi163 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:22:22.668 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:22:22.881 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:22.881 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-03-25T18:22:22.882 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-03-25T18:22:22.882 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-03-25T18:22:23.264 INFO:teuthology.orchestra.run.smithi163.stderr:Generating grub configuration file ... 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:22:23.312 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:22:23.665 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:22:23.803 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.804 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:22:23.804 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-03-25T18:22:23.804 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T18:22:23.804 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T18:22:23.804 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T18:22:23.804 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi130.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T18:22:23.804 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T18:22:23.883 INFO:teuthology.orchestra.run.smithi130.stdout:kernel-5.14.0-432.el9.x86_64 2024-03-25T18:22:23.883 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-03-25T18:22:23.883 DEBUG:teuthology.orchestra.run.smithi130:> sudo rpm -qi grub2-tools 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Name : grub2-tools 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Epoch : 1 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Version : 2.06 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Release : 68.el9 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Architecture: x86_64 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Group : Unspecified 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Size : 8264254 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:License : GPLv3+ 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T18:22:23.920 INFO:teuthology.orchestra.run.smithi130.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Packager : builder@centos.org 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Vendor : CentOS 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Summary : Support tools for GRUB. 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:Description : 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:hardware devices. 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:22:23.921 INFO:teuthology.orchestra.run.smithi130.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T18:22:23.922 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T18:22:23.922 INFO:teuthology.task.kernel:Updating grub on smithi130 to boot 5.14.0-432.el9.x86_64 2024-03-25T18:22:23.922 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T18:22:24.549 INFO:teuthology.orchestra.run.smithi163.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:22:24.574 INFO:teuthology.orchestra.run.smithi130.stderr:Generating grub configuration file ... 2024-03-25T18:22:24.576 INFO:teuthology.orchestra.run.smithi163.stderr:done 2024-03-25T18:22:24.579 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-03-25T18:22:24.593 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.A0PJizifNo 2024-03-25T18:22:24.594 DEBUG:teuthology.orchestra.run.smithi163:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.A0PJizifNo 2024-03-25T18:22:24.662 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.A0PJizifNo 2024-03-25T18:22:24.806 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.A0PJizifNo is 6KB 2024-03-25T18:22:24.854 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.A0PJizifNo 2024-03-25T18:22:24.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:22:24.936 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:22:24.936 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:22:24.936 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:22:24.936 INFO:teuthology.orchestra.run.smithi163.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:22:24.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:22:25.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo shutdown -r now 2024-03-25T18:22:25.922 INFO:teuthology.orchestra.run.smithi130.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T18:22:25.953 INFO:teuthology.orchestra.run.smithi130.stderr:done 2024-03-25T18:22:25.962 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-03-25T18:22:25.976 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.tWNkUwTMCJ 2024-03-25T18:22:25.976 DEBUG:teuthology.orchestra.run.smithi130:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tWNkUwTMCJ 2024-03-25T18:22:26.047 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.tWNkUwTMCJ 2024-03-25T18:22:26.187 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.tWNkUwTMCJ is 6KB 2024-03-25T18:22:26.206 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.tWNkUwTMCJ 2024-03-25T18:22:26.220 DEBUG:teuthology.orchestra.run.smithi130:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T18:22:26.285 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T18:22:26.285 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T18:22:26.285 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T18:22:26.285 INFO:teuthology.orchestra.run.smithi130.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-03-25T18:22:26.287 DEBUG:teuthology.orchestra.run.smithi130:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-03-25T18:22:26.444 DEBUG:teuthology.orchestra.run.smithi130:> sudo shutdown -r now 2024-03-25T18:22:55.097 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T18:22:55.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:22:55.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:22:56.448 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T18:22:56.448 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:22:56.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:23:13.439 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-25T18:23:14.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.130 2024-03-25T18:23:22.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:23:22.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:23:23.977 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:23:23.978 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:23:25.502 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-25T18:23:37.515 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:23:37.515 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:23:40.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-25T18:23:55.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:23:55.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:14.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.163 2024-03-25T18:24:23.991 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T18:24:32.120 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:24:32.121 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:32.808 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-25T18:24:33.331 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:24:33.331 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:24:33.331 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2024-03-25T18:24:33.356 INFO:teuthology.orchestra.run.smithi163.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:24:33.357 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:24:33.357 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:24:33.357 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:24:34.358 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T18:24:34.358 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:24:34.461 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2024-03-25T18:24:34.678 DEBUG:teuthology.parallel:result is None 2024-03-25T18:24:35.992 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:24:35.992 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:24:36.381 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-25T18:24:36.752 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:24:36.752 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-432.el9.x86_64"... 2024-03-25T18:24:36.753 DEBUG:teuthology.orchestra.run.smithi130:> uname -r 2024-03-25T18:24:36.767 INFO:teuthology.orchestra.run.smithi130.stdout:5.14.0-432.el9.x86_64 2024-03-25T18:24:36.768 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-03-25T18:24:36.768 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T18:24:36.768 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T18:24:37.769 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T18:24:37.770 DEBUG:teuthology.orchestra.run.smithi130:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T18:24:38.107 INFO:teuthology.orchestra.run.smithi130.stdout:ttyS1 2024-03-25T18:24:38.148 DEBUG:teuthology.parallel:result is None 2024-03-25T18:24:38.149 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T18:24:38.156 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T18:24:38.156 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:24:38.159 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T18:24:38.177 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T18:24:38.210 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T18:24:38.221 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T18:24:38.221 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:24:38.223 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T18:24:38.255 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T18:24:38.261 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T18:24:38.261 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:24:38.288 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T18:24:38.324 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:24:38.338 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:24:38.379 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:24:38.393 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T18:24:38.395 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T18:24:38.401 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T18:24:38.401 DEBUG:teuthology.orchestra.run.smithi130:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:24:38.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T18:24:38.433 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T18:24:38.458 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T18:24:38.459 DEBUG:teuthology.orchestra.run.smithi130:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:24:38.473 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T18:24:38.490 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:24:38.563 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:24:38.644 DEBUG:teuthology.orchestra.run.smithi130:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:24:38.682 DEBUG:teuthology.orchestra.run.smithi130:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:24:38.729 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T18:24:38.729 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:24:38.796 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:24:38.844 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T18:24:38.891 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:24:38.965 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T18:24:39.019 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T18:24:39.019 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T18:24:39.087 DEBUG:teuthology.orchestra.run.smithi130:> sudo service rsyslog restart 2024-03-25T18:24:39.090 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-03-25T18:24:39.134 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:24:39.173 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:24:39.571 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T18:24:39.579 INFO:teuthology.task.internal:Starting timer... 2024-03-25T18:24:39.580 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T18:24:39.624 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T18:24:39.700 DEBUG:teuthology.orchestra.run.smithi130:> sudo service auditd rotate 2024-03-25T18:24:39.748 INFO:teuthology.orchestra.run.smithi130.stdout:Rotating logs: 2024-03-25T18:24:39.750 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-03-25T18:24:39.820 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-03-25T18:24:39.821 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T18:24:39.821 DEBUG:teuthology.orchestra.run.smithi130:> /usr/sbin/getenforce 2024-03-25T18:24:39.849 INFO:teuthology.orchestra.run.smithi130.stdout:Permissive 2024-03-25T18:24:39.849 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-03-25T18:24:39.870 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-03-25T18:24:39.871 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi130.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-03-25T18:24:39.871 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:24:39.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:24:39.917 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:24:39.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:24:39.944 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T18:24:39.944 DEBUG:teuthology.orchestra.run.smithi130:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:24:40.000 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-03-25T18:24:40.030 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T18:24:40.039 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T18:24:40.046 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T18:24:40.188 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T18:24:40.204 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T18:24:40.205 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi130.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T18:28:54.665 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi130.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-03-25T18:28:54.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:28:54.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi130.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:28:54.759 DEBUG:teuthology.orchestra.run.smithi130:> true 2024-03-25T18:28:54.842 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi130.front.sepia.ceph.com' 2024-03-25T18:28:54.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:28:54.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T18:28:54.932 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-03-25T18:28:55.006 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-03-25T18:28:55.006 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T18:28:55.017 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T18:28:55.017 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:28:55.018 DEBUG:teuthology.orchestra.run.smithi130:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:28:55.020 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T18:28:55.020 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:28:55.051 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:28:55.069 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:28:55.078 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T18:28:55.096 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T18:28:55.098 INFO:teuthology.orchestra.run.smithi130.stderr:sudo: ntpd: command not found 2024-03-25T18:28:55.113 INFO:teuthology.orchestra.run.smithi130.stdout:506 Cannot talk to daemon 2024-03-25T18:28:55.125 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-03-25T18:28:55.129 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:28:55.139 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-03-25T18:28:55.146 INFO:teuthology.orchestra.run.smithi130.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:28:55.156 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T18:28:55.172 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T18:28:55.198 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.203 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.225 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:28:55.229 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:28:55.229 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-03-25T18:28:55.229 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.230 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.230 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.230 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:28:55.230 INFO:teuthology.run_tasks:Running task install... 2024-03-25T18:28:55.239 DEBUG:teuthology.task.install:project ceph 2024-03-25T18:28:55.239 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T18:28:55.239 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T18:28:55.239 INFO:teuthology.task.install:Using flavor: default 2024-03-25T18:28:55.245 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T18:28:55.245 INFO:teuthology.task.install:extra packages: [] 2024-03-25T18:28:55.245 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:28:55.246 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:28:55.248 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T18:28:55.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:28:55.391 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:28:55.391 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:28:55.481 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T18:28:55.481 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T18:28:55.682 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:28:55.682 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T18:28:55.682 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:28:55.710 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:28:55.711 DEBUG:teuthology.orchestra.run.smithi130:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:28:55.778 DEBUG:teuthology.orchestra.run.smithi130:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:28:55.858 DEBUG:teuthology.orchestra.run.smithi130:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:28:55.861 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T18:28:55.862 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T18:28:55.862 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T18:28:55.891 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T18:28:55.891 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T18:28:55.924 INFO:teuthology.orchestra.run.smithi130.stdout:check_obsoletes = 1 2024-03-25T18:28:55.926 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum clean all 2024-03-25T18:28:55.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T18:28:56.041 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T18:28:56.108 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-03-25T18:28:56.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-03-25T18:28:56.168 INFO:teuthology.orchestra.run.smithi130.stdout:74 files removed 2024-03-25T18:28:56.196 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw 2024-03-25T18:28:56.354 INFO:teuthology.orchestra.run.smithi163.stdout:74 files removed 2024-03-25T18:28:56.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw 2024-03-25T18:28:57.403 INFO:teuthology.orchestra.run.smithi130.stdout:ceph packages for x86_64 90 kB/s | 82 kB 00:00 2024-03-25T18:28:57.566 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 92 kB/s | 82 kB 00:00 2024-03-25T18:28:58.162 INFO:teuthology.orchestra.run.smithi130.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T18:28:58.342 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 14 kB/s | 10 kB 00:00 2024-03-25T18:28:58.954 INFO:teuthology.orchestra.run.smithi130.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T18:28:59.170 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T18:28:59.673 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - AppStream 9.8 MB/s | 6.7 MB 00:00 2024-03-25T18:28:59.711 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T18:29:01.487 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS-9 - BaseOS 3.0 MB/s | 2.0 MB 00:00 2024-03-25T18:29:01.498 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS-9 - BaseOS 2.8 MB/s | 2.0 MB 00:00 2024-03-25T18:29:05.958 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T18:29:06.120 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - BaseOS 1.9 MB/s | 8.0 MB 00:04 2024-03-25T18:29:11.556 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - AppStream 4.3 MB/s | 19 MB 00:04 2024-03-25T18:29:11.993 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:04 2024-03-25T18:29:15.547 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T18:29:17.245 INFO:teuthology.orchestra.run.smithi130.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-25T18:29:18.929 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 1.8 MB/s | 6.1 MB 00:03 2024-03-25T18:29:18.997 INFO:teuthology.orchestra.run.smithi130.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T18:29:21.417 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T18:29:22.471 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-03-25T18:29:23.889 INFO:teuthology.orchestra.run.smithi130.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-25T18:29:26.346 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:29:26.359 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrading: 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:29:26.360 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:29:26.361 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout:Installing weak dependencies: 2024-03-25T18:29:26.362 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:Install 30 Packages 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:Upgrade 2 Packages 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 58 M 2024-03-25T18:29:26.363 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:29:27.327 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T18:29:29.772 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:29:29.781 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T18:29:29.782 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T18:29:29.783 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:29:29.784 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout:Install 30 Packages 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 2 Packages 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 58 M 2024-03-25T18:29:29.785 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:29:37.431 INFO:teuthology.orchestra.run.smithi130.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 518 kB/s | 5.3 MB 00:10 2024-03-25T18:29:37.939 INFO:teuthology.orchestra.run.smithi130.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 49 kB/s | 25 kB 00:00 2024-03-25T18:29:39.731 INFO:teuthology.orchestra.run.smithi130.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 414 kB/s | 742 kB 00:01 2024-03-25T18:29:40.631 INFO:teuthology.orchestra.run.smithi130.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 554 kB/s | 498 kB 00:00 2024-03-25T18:29:42.093 INFO:teuthology.orchestra.run.smithi163.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 459 kB/s | 5.3 MB 00:11 2024-03-25T18:29:42.193 INFO:teuthology.orchestra.run.smithi163.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T18:29:43.402 INFO:teuthology.orchestra.run.smithi163.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 614 kB/s | 742 kB 00:01 2024-03-25T18:29:44.177 INFO:teuthology.orchestra.run.smithi163.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 643 kB/s | 498 kB 00:00 2024-03-25T18:29:45.310 INFO:teuthology.orchestra.run.smithi163.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 659 kB/s | 9.7 MB 00:15 2024-03-25T18:29:45.427 INFO:teuthology.orchestra.run.smithi163.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 388 kB/s | 45 kB 00:00 2024-03-25T18:29:45.727 INFO:teuthology.orchestra.run.smithi163.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 438 kB/s | 131 kB 00:00 2024-03-25T18:29:45.897 INFO:teuthology.orchestra.run.smithi130.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 992 kB/s | 5.1 MB 00:05 2024-03-25T18:29:46.006 INFO:teuthology.orchestra.run.smithi130.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 417 kB/s | 45 kB 00:00 2024-03-25T18:29:46.136 INFO:teuthology.orchestra.run.smithi163.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 397 kB/s | 162 kB 00:00 2024-03-25T18:29:46.189 INFO:teuthology.orchestra.run.smithi130.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 717 kB/s | 131 kB 00:00 2024-03-25T18:29:46.356 INFO:teuthology.orchestra.run.smithi130.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 973 kB/s | 162 kB 00:00 2024-03-25T18:29:46.690 INFO:teuthology.orchestra.run.smithi130.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 961 kB/s | 320 kB 00:00 2024-03-25T18:29:46.798 INFO:teuthology.orchestra.run.smithi130.stdout:(10/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 500 kB/s | 9.7 MB 00:19 2024-03-25T18:29:46.811 INFO:teuthology.orchestra.run.smithi163.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 475 kB/s | 320 kB 00:00 2024-03-25T18:29:47.116 INFO:teuthology.orchestra.run.smithi130.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 313 kB/s | 99 kB 00:00 2024-03-25T18:29:47.141 INFO:teuthology.orchestra.run.smithi130.stdout:(12/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 661 kB/s | 298 kB 00:00 2024-03-25T18:29:47.266 INFO:teuthology.orchestra.run.smithi130.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 708 kB/s | 106 kB 00:00 2024-03-25T18:29:47.291 INFO:teuthology.orchestra.run.smithi130.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-25T18:29:47.317 INFO:teuthology.orchestra.run.smithi130.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-25T18:29:47.368 INFO:teuthology.orchestra.run.smithi130.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T18:29:47.392 INFO:teuthology.orchestra.run.smithi130.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T18:29:47.418 INFO:teuthology.orchestra.run.smithi130.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 829 kB/s | 41 kB 00:00 2024-03-25T18:29:47.434 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.436 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.453 INFO:teuthology.orchestra.run.smithi163.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 464 kB/s | 298 kB 00:00 2024-03-25T18:29:47.503 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.514 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.573 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:47.579 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:47.670 INFO:teuthology.orchestra.run.smithi163.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 459 kB/s | 99 kB 00:00 2024-03-25T18:29:47.706 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:47.728 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:47.740 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:47.775 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:47.778 INFO:teuthology.orchestra.run.smithi163.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-25T18:29:47.817 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.820 INFO:teuthology.orchestra.run.smithi163.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T18:29:47.827 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.836 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.844 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:47.845 INFO:teuthology.orchestra.run.smithi163.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T18:29:47.896 INFO:teuthology.orchestra.run.smithi163.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-25T18:29:47.954 INFO:teuthology.orchestra.run.smithi163.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-25T18:29:47.980 INFO:teuthology.orchestra.run.smithi163.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T18:29:47.997 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.004 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.046 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.173 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.233 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:48.349 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.372 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.393 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.413 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.436 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.536 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.561 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:48.675 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:48.700 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:48.749 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:48.751 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:48.767 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:48.786 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:48.823 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:48.868 INFO:teuthology.orchestra.run.smithi130.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 25 kB/s | 36 kB 00:01 2024-03-25T18:29:48.888 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:48.891 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:48.951 INFO:teuthology.orchestra.run.smithi130.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 142 kB/s | 221 kB 00:01 2024-03-25T18:29:48.961 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.018 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:49.029 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:49.075 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.092 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.160 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:49.198 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:49.233 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:49.250 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.270 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.309 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:49.360 INFO:teuthology.orchestra.run.smithi130.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 757 kB/s | 309 kB 00:00 2024-03-25T18:29:49.430 INFO:teuthology.orchestra.run.smithi163.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 152 kB/s | 221 kB 00:01 2024-03-25T18:29:49.440 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:49.461 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:49.494 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:49.510 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:49.591 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:49.671 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:49.677 INFO:teuthology.orchestra.run.smithi130.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-03-25T18:29:49.690 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.744 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.752 INFO:teuthology.orchestra.run.smithi130.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 353 kB/s | 26 kB 00:00 2024-03-25T18:29:49.779 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:49.811 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:49.827 INFO:teuthology.orchestra.run.smithi130.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 647 kB/s | 49 kB 00:00 2024-03-25T18:29:49.839 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:49.866 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.883 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:49.894 INFO:teuthology.orchestra.run.smithi130.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-25T18:29:49.895 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:50.041 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:50.045 INFO:teuthology.orchestra.run.smithi130.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 838 kB 00:00 2024-03-25T18:29:50.129 INFO:teuthology.orchestra.run.smithi130.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-03-25T18:29:50.129 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatlib-service-24.02.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatlib-service-24.02.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:50.163 INFO:teuthology.orchestra.run.smithi163.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 49 kB/s | 36 kB 00:00 2024-03-25T18:29:50.195 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:50.228 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:50.245 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.siena.edu/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 199.223.246.113) 2024-03-25T18:29:50.287 INFO:teuthology.orchestra.run.smithi130.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-25T18:29:50.325 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:50.406 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftp-osl.osuosl.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 140.211.166.134) 2024-03-25T18:29:50.426 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:50.483 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:50.516 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:50.548 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://mirror.team-cymru.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 216.31.0.2) 2024-03-25T18:29:50.610 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:50.627 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://iad.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 72.4.120.219) 2024-03-25T18:29:50.766 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:50.854 INFO:teuthology.orchestra.run.smithi163.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:50.890 INFO:teuthology.orchestra.run.smithi163.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 65 kB/s | 47 kB 00:00 2024-03-25T18:29:51.589 INFO:teuthology.orchestra.run.smithi163.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 441 kB/s | 309 kB 00:00 2024-03-25T18:29:52.130 INFO:teuthology.orchestra.run.smithi163.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 4.4 MB 00:00 2024-03-25T18:29:52.256 INFO:teuthology.orchestra.run.smithi163.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-03-25T18:29:52.372 INFO:teuthology.orchestra.run.smithi163.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 417 kB/s | 49 kB 00:00 2024-03-25T18:29:52.481 INFO:teuthology.orchestra.run.smithi163.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-25T18:29:52.648 INFO:teuthology.orchestra.run.smithi163.stdout:(26/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 617 kB/s | 5.1 MB 00:08 2024-03-25T18:29:52.681 INFO:teuthology.orchestra.run.smithi163.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 838 kB 00:00 2024-03-25T18:29:52.840 INFO:teuthology.orchestra.run.smithi163.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-03-25T18:29:53.257 INFO:teuthology.orchestra.run.smithi163.stdout:(29/32): re2-20211101-3.el9.x86_64.rpm 315 kB/s | 192 kB 00:00 2024-03-25T18:29:54.265 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://ftpmirror.your.org/pub/CentOS-Stream-All/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 163.237.219.7) 2024-03-25T18:29:54.423 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for https://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:54.503 INFO:teuthology.orchestra.run.smithi130.stdout:[MIRROR] qatzip-libs-1.2.0-1.el9.x86_64.rpm: Status code: 404 for http://dfw.mirror.rackspace.com/centos-stream/9-stream/AppStream/x86_64/os/Packages/qatzip-libs-1.2.0-1.el9.x86_64.rpm (IP: 74.205.112.120) 2024-03-25T18:29:54.554 INFO:teuthology.orchestra.run.smithi130.stdout:(29/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 8.3 kB/s | 47 kB 00:05 2024-03-25T18:29:55.390 INFO:teuthology.orchestra.run.smithi130.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 677 kB/s | 3.4 MB 00:05 2024-03-25T18:29:56.774 INFO:teuthology.orchestra.run.smithi163.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 911 kB/s | 3.1 MB 00:03 2024-03-25T18:29:58.329 INFO:teuthology.orchestra.run.smithi130.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 849 kB/s | 3.1 MB 00:03 2024-03-25T18:30:00.861 INFO:teuthology.orchestra.run.smithi163.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 431 kB/s | 3.4 MB 00:08 2024-03-25T18:30:09.973 INFO:teuthology.orchestra.run.smithi163.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 512 kB/s | 20 MB 00:39 2024-03-25T18:30:09.980 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:09.981 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.4 MB/s | 58 MB 00:40 2024-03-25T18:30:10.015 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:30:10.048 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:30:10.049 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:30:10.521 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:30:10.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:30:11.995 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:30:12.076 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:30:12.119 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:30:12.144 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:30:12.167 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:12.183 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:12.326 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:12.380 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:12.703 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:30:12.770 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:12.845 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:12.932 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:30:12.963 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:13.026 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:13.055 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:30:13.098 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:30:13.166 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:13.191 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:13.238 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:13.273 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:30:13.379 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:30:13.407 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:30:13.460 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:30:13.488 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:30:13.517 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:30:13.550 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:30:13.636 INFO:teuthology.orchestra.run.smithi130.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 435 kB/s | 20 MB 00:46 2024-03-25T18:30:13.644 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:13.644 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.2 MB/s | 58 MB 00:47 2024-03-25T18:30:13.675 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:30:13.714 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:30:13.714 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:30:13.861 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:30:13.888 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:13.986 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:14.018 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:30:14.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:30:14.183 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:30:14.183 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:30:14.518 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:30:14.540 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:14.603 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:14.664 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:30:14.692 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:30:15.650 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:30:15.737 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T18:30:15.766 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T18:30:15.800 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T18:30:15.822 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:15.838 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-25T18:30:16.023 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:16.077 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-25T18:30:16.395 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T18:30:16.451 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:16.533 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:16.628 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T18:30:16.652 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:16.698 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T18:30:16.726 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T18:30:16.770 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T18:30:16.816 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:16.838 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:16.863 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:16.863 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:16.927 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T18:30:16.962 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-25T18:30:17.076 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T18:30:17.113 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T18:30:17.153 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T18:30:17.186 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T18:30:17.215 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T18:30:17.248 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T18:30:17.414 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:17.458 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:17.564 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T18:30:17.594 INFO:teuthology.orchestra.run.smithi130.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:17.684 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T18:30:17.687 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:17.687 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:30:17.688 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:30:17.688 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:17.715 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T18:30:17.757 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T18:30:17.801 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:18.212 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T18:30:18.238 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:18.301 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T18:30:18.369 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T18:30:18.407 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T18:30:20.570 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:20.613 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:21.167 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T18:30:21.199 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:21.279 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T18:30:21.280 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T18:30:21.280 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T18:30:21.280 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:21.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-03-25T18:30:27.022 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-03-25T18:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-03-25T18:30:27.023 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:27.932 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:27.956 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:27.958 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:29.160 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:29.161 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:30:29.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:30:29.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:30:30.150 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:30.150 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:30.150 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.151 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.152 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:30.153 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:30:30.428 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /sys 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /proc 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /mnt 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /var/tmp 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /home 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /root 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout:skipping the directory /tmp 2024-03-25T18:30:30.475 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:30.987 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:01:03 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:30:31.140 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:30:31.140 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:31.141 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 48 M 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 204 M 2024-03-25T18:30:31.142 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:30:31.342 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T18:30:31.366 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:31.368 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:31.449 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-25T18:30:31.460 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T18:30:31.461 INFO:teuthology.orchestra.run.smithi130.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:31.499 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T18:30:31.532 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T18:30:31.666 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T18:30:32.000 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 226 kB/s | 163 kB 00:00 2024-03-25T18:30:32.586 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:32.586 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T18:30:32.586 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T18:30:32.586 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T18:30:32.587 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T18:30:32.588 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T18:30:32.589 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T18:30:32.589 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T18:30:33.550 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout:Upgraded: 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T18:30:33.551 INFO:teuthology.orchestra.run.smithi130.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.552 INFO:teuthology.orchestra.run.smithi130.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:33.553 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:30:33.952 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-test 2024-03-25T18:30:34.512 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:01:11 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:30:34.668 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T18:30:34.669 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout:Install 6 Packages 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:30:34.670 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 48 M 2024-03-25T18:30:34.671 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 204 M 2024-03-25T18:30:34.671 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:30:34.865 INFO:teuthology.orchestra.run.smithi130.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T18:30:34.915 INFO:teuthology.orchestra.run.smithi130.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-25T18:30:34.940 INFO:teuthology.orchestra.run.smithi130.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-25T18:30:35.191 INFO:teuthology.orchestra.run.smithi130.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 748 kB/s | 187 kB 00:00 2024-03-25T18:30:35.632 INFO:teuthology.orchestra.run.smithi130.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 176 kB/s | 163 kB 00:00 2024-03-25T18:32:12.953 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 476 kB/s | 47 MB 01:41 2024-03-25T18:32:12.955 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:12.955 INFO:teuthology.orchestra.run.smithi163.stdout:Total 485 kB/s | 48 MB 01:41 2024-03-25T18:32:12.966 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:32:12.985 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:32:12.985 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:32:13.378 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:32:13.378 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:32:13.967 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:32:14.036 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:32:14.073 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:32:14.113 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:32:14.158 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:32:14.189 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:14.855 INFO:teuthology.orchestra.run.smithi130.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 483 kB/s | 47 MB 01:40 2024-03-25T18:32:14.857 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:14.857 INFO:teuthology.orchestra.run.smithi130.stdout:Total 492 kB/s | 48 MB 01:40 2024-03-25T18:32:14.868 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:32:14.888 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:32:14.888 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:32:15.269 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:32:15.269 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:32:15.850 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:32:15.929 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T18:32:15.969 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T18:32:16.010 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:32:16.058 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T18:32:16.082 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:18.741 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:18.814 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:32:20.539 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T18:32:20.623 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:32:21.156 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:32:21.156 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:32:21.156 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:32:21.156 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:32:21.157 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:32:21.157 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:32:21.542 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:32:21.543 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:32:21.543 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:32:21.543 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:32:21.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph 2024-03-25T18:32:22.419 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:02:55 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:32:22.587 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:32:22.589 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-25T18:32:22.589 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:32:22.590 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:32:22.591 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:32:22.593 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:32:22.593 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:32:22.593 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:32:22.594 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout:========================================================================================= 2024-03-25T18:32:22.595 INFO:teuthology.orchestra.run.smithi163.stdout:Install 36 Packages 2024-03-25T18:32:22.596 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:32:22.596 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 30 M 2024-03-25T18:32:22.596 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 107 M 2024-03-25T18:32:22.596 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T18:32:22.983 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T18:32:23.308 INFO:teuthology.orchestra.run.smithi163.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T18:32:23.366 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T18:32:23.366 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: jq-1.6-16.el9.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:32:23.367 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:32:23.659 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph 2024-03-25T18:32:24.232 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:03:01 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:32:24.385 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================= 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================= 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T18:32:24.388 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T18:32:24.389 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T18:32:24.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout:========================================================================================= 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout:Install 36 Packages 2024-03-25T18:32:24.391 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:32:24.392 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 30 M 2024-03-25T18:32:24.392 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 107 M 2024-03-25T18:32:24.392 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:32:25.217 INFO:teuthology.orchestra.run.smithi130.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T18:32:26.308 INFO:teuthology.orchestra.run.smithi163.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 443 kB/s | 1.5 MB 00:03 2024-03-25T18:32:28.617 INFO:teuthology.orchestra.run.smithi130.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 404 kB/s | 1.5 MB 00:03 2024-03-25T18:32:29.166 INFO:teuthology.orchestra.run.smithi163.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 345 kB/s | 2.1 MB 00:06 2024-03-25T18:32:29.284 INFO:teuthology.orchestra.run.smithi130.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 489 kB/s | 2.1 MB 00:04 2024-03-25T18:32:29.892 INFO:teuthology.orchestra.run.smithi163.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 341 kB/s | 247 kB 00:00 2024-03-25T18:32:30.000 INFO:teuthology.orchestra.run.smithi163.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 187 kB 00:00 2024-03-25T18:32:30.017 INFO:teuthology.orchestra.run.smithi130.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 337 kB/s | 247 kB 00:00 2024-03-25T18:32:30.034 INFO:teuthology.orchestra.run.smithi163.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T18:32:30.059 INFO:teuthology.orchestra.run.smithi163.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:32:30.084 INFO:teuthology.orchestra.run.smithi163.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:32:30.142 INFO:teuthology.orchestra.run.smithi130.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T18:32:30.192 INFO:teuthology.orchestra.run.smithi130.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-25T18:32:30.209 INFO:teuthology.orchestra.run.smithi163.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 309 kB/s | 39 kB 00:00 2024-03-25T18:32:30.218 INFO:teuthology.orchestra.run.smithi130.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T18:32:30.243 INFO:teuthology.orchestra.run.smithi130.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T18:32:30.251 INFO:teuthology.orchestra.run.smithi163.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:32:30.268 INFO:teuthology.orchestra.run.smithi130.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T18:32:30.310 INFO:teuthology.orchestra.run.smithi130.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T18:32:30.319 INFO:teuthology.orchestra.run.smithi163.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T18:32:30.369 INFO:teuthology.orchestra.run.smithi163.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-25T18:32:30.386 INFO:teuthology.orchestra.run.smithi130.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T18:32:30.403 INFO:teuthology.orchestra.run.smithi163.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T18:32:30.419 INFO:teuthology.orchestra.run.smithi130.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T18:32:30.444 INFO:teuthology.orchestra.run.smithi163.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:32:30.453 INFO:teuthology.orchestra.run.smithi130.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T18:32:30.495 INFO:teuthology.orchestra.run.smithi130.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T18:32:30.520 INFO:teuthology.orchestra.run.smithi163.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.6 MB/s | 125 kB 00:00 2024-03-25T18:32:30.603 INFO:teuthology.orchestra.run.smithi163.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 268 kB/s | 22 kB 00:00 2024-03-25T18:32:30.687 INFO:teuthology.orchestra.run.smithi130.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 652 kB/s | 125 kB 00:00 2024-03-25T18:32:30.787 INFO:teuthology.orchestra.run.smithi130.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 223 kB/s | 22 kB 00:00 2024-03-25T18:32:30.821 INFO:teuthology.orchestra.run.smithi163.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 697 kB/s | 151 kB 00:00 2024-03-25T18:32:30.879 INFO:teuthology.orchestra.run.smithi163.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 746 kB/s | 43 kB 00:00 2024-03-25T18:32:30.971 INFO:teuthology.orchestra.run.smithi163.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-03-25T18:32:31.046 INFO:teuthology.orchestra.run.smithi130.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-25T18:32:31.072 INFO:teuthology.orchestra.run.smithi163.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T18:32:31.104 INFO:teuthology.orchestra.run.smithi130.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-03-25T18:32:31.130 INFO:teuthology.orchestra.run.smithi163.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-25T18:32:31.189 INFO:teuthology.orchestra.run.smithi163.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T18:32:31.205 INFO:teuthology.orchestra.run.smithi130.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T18:32:31.248 INFO:teuthology.orchestra.run.smithi163.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 397 kB/s | 23 kB 00:00 2024-03-25T18:32:31.296 INFO:teuthology.orchestra.run.smithi130.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T18:32:31.306 INFO:teuthology.orchestra.run.smithi163.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 334 kB/s | 19 kB 00:00 2024-03-25T18:32:31.355 INFO:teuthology.orchestra.run.smithi130.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-25T18:32:31.365 INFO:teuthology.orchestra.run.smithi163.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T18:32:31.413 INFO:teuthology.orchestra.run.smithi130.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 303 kB/s | 18 kB 00:00 2024-03-25T18:32:31.423 INFO:teuthology.orchestra.run.smithi163.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-25T18:32:31.472 INFO:teuthology.orchestra.run.smithi130.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-03-25T18:32:31.482 INFO:teuthology.orchestra.run.smithi163.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:32:31.531 INFO:teuthology.orchestra.run.smithi130.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 335 kB/s | 19 kB 00:00 2024-03-25T18:32:31.549 INFO:teuthology.orchestra.run.smithi163.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T18:32:31.589 INFO:teuthology.orchestra.run.smithi130.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 338 kB/s | 20 kB 00:00 2024-03-25T18:32:31.616 INFO:teuthology.orchestra.run.smithi163.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-03-25T18:32:31.656 INFO:teuthology.orchestra.run.smithi130.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 694 kB/s | 46 kB 00:00 2024-03-25T18:32:31.675 INFO:teuthology.orchestra.run.smithi163.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:32:31.715 INFO:teuthology.orchestra.run.smithi130.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T18:32:31.733 INFO:teuthology.orchestra.run.smithi163.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-25T18:32:31.801 INFO:teuthology.orchestra.run.smithi163.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:32:31.815 INFO:teuthology.orchestra.run.smithi130.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-25T18:32:31.876 INFO:teuthology.orchestra.run.smithi163.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-25T18:32:31.882 INFO:teuthology.orchestra.run.smithi130.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 246 kB/s | 16 kB 00:00 2024-03-25T18:32:31.926 INFO:teuthology.orchestra.run.smithi163.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 401 kB/s | 20 kB 00:00 2024-03-25T18:32:31.941 INFO:teuthology.orchestra.run.smithi130.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T18:32:31.999 INFO:teuthology.orchestra.run.smithi130.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 613 kB/s | 36 kB 00:00 2024-03-25T18:32:32.066 INFO:teuthology.orchestra.run.smithi130.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T18:32:32.158 INFO:teuthology.orchestra.run.smithi130.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.6 MB/s | 427 kB 00:00 2024-03-25T18:32:32.217 INFO:teuthology.orchestra.run.smithi130.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 344 kB/s | 20 kB 00:00 2024-03-25T18:32:34.393 INFO:teuthology.orchestra.run.smithi163.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 433 kB/s | 4.7 MB 00:11 2024-03-25T18:32:35.542 INFO:teuthology.orchestra.run.smithi130.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 465 kB/s | 4.7 MB 00:10 2024-03-25T18:33:09.438 INFO:teuthology.orchestra.run.smithi130.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 421 kB/s | 17 MB 00:40 2024-03-25T18:33:09.443 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:09.443 INFO:teuthology.orchestra.run.smithi130.stdout:Total 674 kB/s | 30 MB 00:45 2024-03-25T18:33:09.467 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:33:09.509 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:33:09.509 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:33:09.911 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:33:09.912 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:33:11.250 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:33:11.307 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:33:11.352 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:33:11.384 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:33:11.454 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:33:11.499 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:33:11.540 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:33:11.631 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:33:11.718 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:33:11.768 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:33:11.840 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:33:11.894 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:33:11.932 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:33:11.981 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:33:12.037 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:33:12.221 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:33:12.269 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:33:12.299 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:33:12.342 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:33:12.373 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:33:12.447 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:33:12.483 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:12.516 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:33:12.557 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:33:12.592 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:33:12.626 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:33:12.717 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:33:12.765 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:33:12.846 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:33:12.991 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:33:13.110 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:13.278 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:33:13.308 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:13.334 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:14.768 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:14.794 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:14.794 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:14.794 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:33:14.794 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:14.794 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:14.795 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:15.235 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:15.258 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:15.460 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:15.483 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:15.483 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:15.483 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:33:15.483 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:15.484 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:15.484 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:15.549 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:17.285 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:17.285 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:33:17.285 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:33:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:33:17.287 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:33:17.288 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:33:17.462 INFO:teuthology.orchestra.run.smithi163.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 336 kB/s | 17 MB 00:51 2024-03-25T18:33:17.466 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:17.466 INFO:teuthology.orchestra.run.smithi163.stdout:Total 554 kB/s | 30 MB 00:54 2024-03-25T18:33:17.490 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:33:17.532 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:33:17.532 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:33:17.967 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:33:17.967 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:18.086 INFO:teuthology.orchestra.run.smithi130.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:33:18.087 INFO:teuthology.orchestra.run.smithi130.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:33:18.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:33:18.089 INFO:teuthology.orchestra.run.smithi130.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:33:18.089 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:18.089 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:18.457 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base 2024-03-25T18:33:19.040 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:19.138 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:19.190 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:33:19.191 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:19.263 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install cephadm 2024-03-25T18:33:19.283 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:33:19.348 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T18:33:19.384 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T18:33:19.417 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T18:33:19.489 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T18:33:19.533 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T18:33:19.565 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T18:33:19.657 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T18:33:19.736 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T18:33:19.777 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T18:33:19.819 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:03:56 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:19.849 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T18:33:19.903 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T18:33:19.941 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T18:33:19.970 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:33:19.971 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 747 k 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 754 k 2024-03-25T18:33:19.972 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:33:19.989 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T18:33:20.045 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T18:33:20.240 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T18:33:20.287 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T18:33:20.326 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T18:33:20.361 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T18:33:20.390 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T18:33:20.468 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T18:33:20.501 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:20.534 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T18:33:20.574 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T18:33:20.610 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T18:33:20.644 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T18:33:20.733 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T18:33:20.767 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T18:33:20.840 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T18:33:20.978 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T18:33:21.102 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:21.269 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T18:33:21.298 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:21.323 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T18:33:21.323 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:21.324 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T18:33:21.324 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:21.324 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T18:33:21.324 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:21.542 INFO:teuthology.orchestra.run.smithi130.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 476 kB/s | 747 kB 00:01 2024-03-25T18:33:21.543 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:21.543 INFO:teuthology.orchestra.run.smithi130.stdout:Total 476 kB/s | 747 kB 00:01 2024-03-25T18:33:21.544 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:33:21.549 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:33:21.550 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:33:21.555 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:33:21.556 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:33:21.630 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:33:22.028 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:22.123 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:22.534 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:22.765 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T18:33:22.792 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:22.816 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:23.035 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:33:23.233 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T18:33:23.256 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:23.475 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T18:33:23.499 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:23.565 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:23.596 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:33:23.749 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 145 k 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 436 k 2024-03-25T18:33:23.750 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:33:24.406 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 222 kB/s | 145 kB 00:00 2024-03-25T18:33:24.407 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:24.407 INFO:teuthology.orchestra.run.smithi130.stdout:Total 221 kB/s | 145 kB 00:00 2024-03-25T18:33:24.407 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:33:24.414 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:33:24.414 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:33:24.467 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:33:24.467 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:33:24.617 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:33:24.673 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:24.697 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:24.697 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:24.697 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:33:24.697 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T18:33:25.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T18:33:25.326 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T18:33:25.327 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:25.667 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:25.893 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr 2024-03-25T18:33:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T18:33:26.286 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T18:33:26.287 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T18:33:26.288 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:26.289 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:26.468 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:03 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:26.565 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:26.617 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:26.618 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:33:26.618 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:26.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base 2024-03-25T18:33:26.684 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:33:27.193 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:27.245 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:27.290 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:27.342 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:27.343 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:33:27.343 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:27.394 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:27.395 INFO:teuthology.orchestra.run.smithi130.stdout:====================================================================================== 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout:====================================================================================== 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:33:27.396 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout:====================================================================================== 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout:Install 12 Packages 2024-03-25T18:33:27.397 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:27.398 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 6.9 M 2024-03-25T18:33:27.398 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 26 M 2024-03-25T18:33:27.398 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:33:27.420 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install cephadm 2024-03-25T18:33:27.984 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:28.134 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:33:28.135 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:28.136 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:33:28.136 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:28.136 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 747 k 2024-03-25T18:33:28.136 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 754 k 2024-03-25T18:33:28.136 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:33:28.401 INFO:teuthology.orchestra.run.smithi130.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T18:33:28.426 INFO:teuthology.orchestra.run.smithi130.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T18:33:28.576 INFO:teuthology.orchestra.run.smithi130.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T18:33:28.610 INFO:teuthology.orchestra.run.smithi130.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-25T18:33:29.077 INFO:teuthology.orchestra.run.smithi130.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T18:33:29.110 INFO:teuthology.orchestra.run.smithi130.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.6 MB/s | 863 kB 00:00 2024-03-25T18:33:29.152 INFO:teuthology.orchestra.run.smithi130.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 259 kB/s | 19 kB 00:00 2024-03-25T18:33:29.236 INFO:teuthology.orchestra.run.smithi130.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.7 MB/s | 144 kB 00:00 2024-03-25T18:33:29.327 INFO:teuthology.orchestra.run.smithi130.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-25T18:33:29.411 INFO:teuthology.orchestra.run.smithi130.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.2 MB/s | 188 kB 00:00 2024-03-25T18:33:29.500 INFO:teuthology.orchestra.run.smithi163.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 548 kB/s | 747 kB 00:01 2024-03-25T18:33:29.501 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:29.501 INFO:teuthology.orchestra.run.smithi163.stdout:Total 547 kB/s | 747 kB 00:01 2024-03-25T18:33:29.502 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:33:29.508 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:33:29.508 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:33:29.514 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:33:29.515 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:33:29.588 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:33:29.686 INFO:teuthology.orchestra.run.smithi130.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-03-25T18:33:30.104 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.198 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.596 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:30.882 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:31.097 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T18:33:31.672 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:31.826 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:31.827 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:31.827 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:33:31.827 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 145 k 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 436 k 2024-03-25T18:33:31.828 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:33:32.566 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 197 kB/s | 145 kB 00:00 2024-03-25T18:33:32.566 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:32.567 INFO:teuthology.orchestra.run.smithi163.stdout:Total 197 kB/s | 145 kB 00:00 2024-03-25T18:33:32.567 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:33:32.575 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:33:32.575 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:33:32.629 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:33:32.629 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:33:32.769 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:33:32.834 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:32.858 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:32.858 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:33:32.859 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T18:33:32.859 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:33.552 INFO:teuthology.orchestra.run.smithi130.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 321 kB/s | 1.8 MB 00:05 2024-03-25T18:33:33.559 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:33.559 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T18:33:33.579 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:33:33.598 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:33:33.598 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:33:33.772 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:33:33.772 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:33.846 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:34.112 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr 2024-03-25T18:33:34.284 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:33:34.420 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:33:34.471 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:33:34.526 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:33:34.565 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:33:34.673 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:33:34.685 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:07 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:34.784 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:33:34.797 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:33:34.836 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:34.837 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:33:34.837 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:34.882 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:33:34.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T18:33:34.927 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:33:34.964 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:34.995 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:33:35.237 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:33:35.271 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:35.472 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:08 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:35.622 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T18:33:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:35.625 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:33:35.626 INFO:teuthology.orchestra.run.smithi163.stdout:====================================================================================== 2024-03-25T18:33:35.626 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-03-25T18:33:35.626 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:35.627 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 6.9 M 2024-03-25T18:33:35.627 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-03-25T18:33:35.627 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:33:36.043 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:36.115 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:33:36.115 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:33:36.115 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:36.116 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:33:36.117 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:33:36.381 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:36.382 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:33:36.472 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 29 kB/s | 15 kB 00:00 2024-03-25T18:33:36.497 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 56 kB/s | 31 kB 00:00 2024-03-25T18:33:36.645 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:33:36.647 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T18:33:36.681 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-25T18:33:36.798 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 5.6 MB/s | 863 kB 00:00 2024-03-25T18:33:37.031 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-25T18:33:37.206 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:33:37.207 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T18:33:37.307 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.4 MB/s | 144 kB 00:00 2024-03-25T18:33:37.357 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:33:37.358 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:33:37.359 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout:============================================================================================ 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout:Install 12 Packages 2024-03-25T18:33:37.360 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:33:37.361 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 37 M 2024-03-25T18:33:37.361 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 209 M 2024-03-25T18:33:37.361 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:33:37.399 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 335 kB/s | 31 kB 00:00 2024-03-25T18:33:37.499 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-25T18:33:37.616 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.5 MB/s | 2.0 MB 00:00 2024-03-25T18:33:37.624 INFO:teuthology.orchestra.run.smithi130.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 281 kB/s | 32 kB 00:00 2024-03-25T18:33:37.683 INFO:teuthology.orchestra.run.smithi130.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 485 kB/s | 17 kB 00:00 2024-03-25T18:33:37.725 INFO:teuthology.orchestra.run.smithi130.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 821 kB/s | 34 kB 00:00 2024-03-25T18:33:37.933 INFO:teuthology.orchestra.run.smithi130.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2024-03-25T18:33:38.000 INFO:teuthology.orchestra.run.smithi130.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 6.1 MB/s | 3.0 MB 00:00 2024-03-25T18:33:38.101 INFO:teuthology.orchestra.run.smithi130.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T18:33:38.126 INFO:teuthology.orchestra.run.smithi130.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.7 MB/s | 217 kB 00:00 2024-03-25T18:33:38.318 INFO:teuthology.orchestra.run.smithi130.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-03-25T18:33:38.360 INFO:teuthology.orchestra.run.smithi130.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.9 MB/s | 195 kB 00:00 2024-03-25T18:33:38.518 INFO:teuthology.orchestra.run.smithi130.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-25T18:33:38.777 INFO:teuthology.orchestra.run.smithi130.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 591 kB/s | 245 kB 00:00 2024-03-25T18:33:39.941 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 456 kB/s | 1.8 MB 00:03 2024-03-25T18:33:39.946 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:39.946 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.6 MB/s | 6.9 MB 00:04 2024-03-25T18:33:39.966 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:33:39.986 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:33:39.986 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:33:40.161 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:33:40.162 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:33:40.683 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:33:40.767 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T18:33:40.803 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T18:33:40.850 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T18:33:40.888 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T18:33:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T18:33:41.122 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T18:33:41.217 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T18:33:41.264 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T18:33:41.297 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:41.320 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T18:33:41.555 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T18:33:41.587 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T18:33:42.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T18:33:42.324 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T18:33:42.687 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:42.688 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:33:42.956 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T18:33:43.515 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:33:43.666 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:33:43.667 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-25T18:33:43.667 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:33:43.667 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-25T18:33:43.667 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T18:33:43.668 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================================ 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout:Install 12 Packages 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 37 M 2024-03-25T18:33:43.669 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 209 M 2024-03-25T18:33:43.670 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:33:44.050 INFO:teuthology.orchestra.run.smithi163.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 217 kB/s | 32 kB 00:00 2024-03-25T18:33:44.092 INFO:teuthology.orchestra.run.smithi163.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 413 kB/s | 17 kB 00:00 2024-03-25T18:33:44.150 INFO:teuthology.orchestra.run.smithi163.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 586 kB/s | 34 kB 00:00 2024-03-25T18:33:45.259 INFO:teuthology.orchestra.run.smithi163.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 4.3 MB/s | 4.7 MB 00:01 2024-03-25T18:33:45.418 INFO:teuthology.orchestra.run.smithi163.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-03-25T18:33:45.751 INFO:teuthology.orchestra.run.smithi163.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 653 kB/s | 217 kB 00:00 2024-03-25T18:33:45.993 INFO:teuthology.orchestra.run.smithi163.stdout:(7/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.5 MB/s | 4.8 MB 00:00 2024-03-25T18:33:47.452 INFO:teuthology.orchestra.run.smithi163.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.2 MB/s | 16 MB 00:01 2024-03-25T18:33:48.085 INFO:teuthology.orchestra.run.smithi163.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 384 kB/s | 803 kB 00:02 2024-03-25T18:33:48.110 INFO:teuthology.orchestra.run.smithi163.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 297 kB/s | 195 kB 00:00 2024-03-25T18:33:48.228 INFO:teuthology.orchestra.run.smithi163.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.7 MB/s | 245 kB 00:00 2024-03-25T18:34:01.174 INFO:teuthology.orchestra.run.smithi130.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 320 kB/s | 7.4 MB 00:23 2024-03-25T18:34:01.176 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:01.177 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.6 MB/s | 37 MB 00:23 2024-03-25T18:34:01.219 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:01.245 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:01.245 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:01.614 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:01.615 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:02.339 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:02.414 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:34:02.445 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:34:02.536 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:02.582 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:34:02.757 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:02.789 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:02.870 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:34:03.505 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:34:03.607 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:34:04.217 INFO:teuthology.orchestra.run.smithi163.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 372 kB/s | 7.4 MB 00:20 2024-03-25T18:34:04.219 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:04.220 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.8 MB/s | 37 MB 00:20 2024-03-25T18:34:04.263 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:04.289 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:04.289 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:04.649 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:04.649 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:04.830 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:34:05.352 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:05.423 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T18:34:05.454 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T18:34:05.548 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:05.599 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T18:34:05.768 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:05.790 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:05.857 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:34:05.877 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T18:34:05.886 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:06.550 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T18:34:06.670 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T18:34:07.917 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T18:34:08.599 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:08.599 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:34:08.599 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:34:08.600 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:34:08.950 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:34:09.071 INFO:teuthology.orchestra.run.smithi130.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:09.072 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:09.192 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:09.499 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:34:10.073 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:10.224 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:10.225 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:10.225 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:34:10.225 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:10.225 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:10.225 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:34:10.226 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout:Install 13 Packages 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:10.227 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 2.0 M 2024-03-25T18:34:10.228 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 26 M 2024-03-25T18:34:10.228 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:10.544 INFO:teuthology.orchestra.run.smithi130.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 187 kB/s | 24 kB 00:00 2024-03-25T18:34:10.569 INFO:teuthology.orchestra.run.smithi130.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 116 kB/s | 18 kB 00:00 2024-03-25T18:34:10.636 INFO:teuthology.orchestra.run.smithi130.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.9 MB/s | 174 kB 00:00 2024-03-25T18:34:10.662 INFO:teuthology.orchestra.run.smithi130.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 132 kB 00:00 2024-03-25T18:34:10.695 INFO:teuthology.orchestra.run.smithi130.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T18:34:10.720 INFO:teuthology.orchestra.run.smithi130.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 731 kB/s | 43 kB 00:00 2024-03-25T18:34:10.871 INFO:teuthology.orchestra.run.smithi130.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 100 kB/s | 15 kB 00:00 2024-03-25T18:34:10.896 INFO:teuthology.orchestra.run.smithi130.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 161 kB/s | 32 kB 00:00 2024-03-25T18:34:10.921 INFO:teuthology.orchestra.run.smithi130.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 103 kB/s | 52 kB 00:00 2024-03-25T18:34:11.055 INFO:teuthology.orchestra.run.smithi130.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 1.1 MB/s | 203 kB 00:00 2024-03-25T18:34:11.080 INFO:teuthology.orchestra.run.smithi130.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 372 kB/s | 59 kB 00:00 2024-03-25T18:34:11.122 INFO:teuthology.orchestra.run.smithi130.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-03-25T18:34:11.155 INFO:teuthology.orchestra.run.smithi130.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.9 MB/s | 1.0 MB 00:00 2024-03-25T18:34:11.157 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:11.158 INFO:teuthology.orchestra.run.smithi130.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T18:34:11.158 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:11.171 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:11.171 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:11.262 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:11.264 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T18:34:11.473 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T18:34:11.474 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T18:34:11.772 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:11.919 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T18:34:11.919 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:11.920 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:11.952 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:34:12.006 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:34:12.078 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:34:12.116 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:34:12.157 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:34:12.251 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:34:12.332 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-rook 2024-03-25T18:34:12.351 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:34:12.416 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:34:12.905 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:13.056 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:13.057 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:13.057 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T18:34:13.058 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout:Install 13 Packages 2024-03-25T18:34:13.059 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:13.060 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 2.0 M 2024-03-25T18:34:13.060 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 26 M 2024-03-25T18:34:13.060 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:13.135 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:34:13.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:34:13.231 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:34:13.279 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:34:13.301 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:13.520 INFO:teuthology.orchestra.run.smithi163.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 226 kB/s | 24 kB 00:00 2024-03-25T18:34:13.571 INFO:teuthology.orchestra.run.smithi163.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-25T18:34:13.621 INFO:teuthology.orchestra.run.smithi163.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.2 MB/s | 174 kB 00:00 2024-03-25T18:34:13.646 INFO:teuthology.orchestra.run.smithi163.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T18:34:13.671 INFO:teuthology.orchestra.run.smithi163.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T18:34:13.697 INFO:teuthology.orchestra.run.smithi163.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 861 kB/s | 43 kB 00:00 2024-03-25T18:34:13.831 INFO:teuthology.orchestra.run.smithi163.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-25T18:34:13.856 INFO:teuthology.orchestra.run.smithi163.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 94 kB/s | 15 kB 00:00 2024-03-25T18:34:13.956 INFO:teuthology.orchestra.run.smithi163.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.6 MB/s | 203 kB 00:00 2024-03-25T18:34:14.015 INFO:teuthology.orchestra.run.smithi163.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-03-25T18:34:14.063 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:14.063 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:34:14.063 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:34:14.063 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:34:14.064 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:34:14.074 INFO:teuthology.orchestra.run.smithi163.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-03-25T18:34:14.149 INFO:teuthology.orchestra.run.smithi163.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-25T18:34:14.257 INFO:teuthology.orchestra.run.smithi163.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 62 kB/s | 52 kB 00:00 2024-03-25T18:34:14.259 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:14.259 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-25T18:34:14.260 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:14.273 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:14.273 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:14.362 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:14.363 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:14.419 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:34:14.419 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:34:14.420 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:34:14.421 INFO:teuthology.orchestra.run.smithi130.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:34:14.421 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:14.421 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:14.676 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:34:14.881 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:15.035 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T18:34:15.087 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T18:34:15.147 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T18:34:15.184 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T18:34:15.225 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T18:34:15.238 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:15.319 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T18:34:15.389 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:34:15.390 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:Install 6 Packages 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 6.8 M 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 33 M 2024-03-25T18:34:15.391 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:15.427 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T18:34:15.486 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T18:34:15.923 INFO:teuthology.orchestra.run.smithi130.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 862 kB/s | 248 kB 00:00 2024-03-25T18:34:16.006 INFO:teuthology.orchestra.run.smithi130.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 5.8 MB 00:00 2024-03-25T18:34:16.206 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T18:34:16.258 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T18:34:16.292 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T18:34:16.340 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T18:34:16.354 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:16.498 INFO:teuthology.orchestra.run.smithi130.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 117 kB/s | 58 kB 00:00 2024-03-25T18:34:16.665 INFO:teuthology.orchestra.run.smithi130.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 306 kB/s | 51 kB 00:00 2024-03-25T18:34:16.724 INFO:teuthology.orchestra.run.smithi130.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 685 kB/s | 548 kB 00:00 2024-03-25T18:34:16.874 INFO:teuthology.orchestra.run.smithi130.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 111 kB/s | 138 kB 00:01 2024-03-25T18:34:16.877 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:16.877 INFO:teuthology.orchestra.run.smithi130.stdout:Total 4.6 MB/s | 6.8 MB 00:01 2024-03-25T18:34:16.891 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:16.902 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:16.902 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:16.991 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:16.993 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:17.162 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T18:34:17.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T18:34:17.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T18:34:17.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T18:34:17.162 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T18:34:17.163 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T18:34:17.286 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:17.471 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:34:17.526 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T18:34:17.526 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:17.526 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:17.526 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:17.527 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:17.533 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:34:17.774 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T18:34:18.050 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:34:18.135 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:34:18.197 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:34:18.220 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:18.339 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:18.488 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:18.489 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:18.489 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:34:18.489 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:18.489 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout:Install 6 Packages 2024-03-25T18:34:18.490 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:18.491 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 6.8 M 2024-03-25T18:34:18.491 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 33 M 2024-03-25T18:34:18.491 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:19.010 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:19.010 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:34:19.010 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:34:19.011 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:34:19.011 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:34:19.011 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:34:19.187 INFO:teuthology.orchestra.run.smithi163.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T18:34:19.212 INFO:teuthology.orchestra.run.smithi163.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 834 kB/s | 248 kB 00:00 2024-03-25T18:34:19.334 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:34:19.334 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:19.334 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:19.334 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:19.334 INFO:teuthology.orchestra.run.smithi130.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:19.335 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:19.380 INFO:teuthology.orchestra.run.smithi163.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 346 kB/s | 58 kB 00:00 2024-03-25T18:34:19.455 INFO:teuthology.orchestra.run.smithi163.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 680 kB/s | 51 kB 00:00 2024-03-25T18:34:19.488 INFO:teuthology.orchestra.run.smithi163.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.8 MB/s | 548 kB 00:00 2024-03-25T18:34:19.586 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-fuse 2024-03-25T18:34:19.672 INFO:teuthology.orchestra.run.smithi163.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 182 kB/s | 138 kB 00:00 2024-03-25T18:34:19.673 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:19.673 INFO:teuthology.orchestra.run.smithi163.stdout:Total 5.7 MB/s | 6.8 MB 00:01 2024-03-25T18:34:19.686 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:19.697 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:19.697 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:19.786 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:19.787 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:20.094 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:20.145 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:20.264 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T18:34:20.294 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:20.294 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout:Install 2 Packages 2024-03-25T18:34:20.295 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:20.296 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 964 k 2024-03-25T18:34:20.296 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.8 M 2024-03-25T18:34:20.296 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:20.335 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T18:34:20.417 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 653 kB/s | 79 kB 00:00 2024-03-25T18:34:20.858 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T18:34:20.951 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T18:34:21.013 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T18:34:21.036 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:21.831 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T18:34:21.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T18:34:21.831 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T18:34:21.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T18:34:21.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T18:34:21.832 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:22.140 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:22.388 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse 2024-03-25T18:34:22.808 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 352 kB/s | 885 kB 00:02 2024-03-25T18:34:22.810 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:22.810 INFO:teuthology.orchestra.run.smithi130.stdout:Total 383 kB/s | 964 kB 00:02 2024-03-25T18:34:22.810 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:22.835 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:22.835 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:22.917 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:22.917 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:22.952 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:23.100 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:23.101 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:23.101 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:23.101 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:34:23.101 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-25T18:34:23.102 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:23.103 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 964 k 2024-03-25T18:34:23.103 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.8 M 2024-03-25T18:34:23.103 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:23.242 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 567 kB/s | 79 kB 00:00 2024-03-25T18:34:23.251 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:34:23.325 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:24.079 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:24.080 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:34:24.398 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:34:24.398 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:24.399 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:24.399 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:34:24.399 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:24.399 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:24.683 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-volume 2024-03-25T18:34:25.017 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 462 kB/s | 885 kB 00:01 2024-03-25T18:34:25.019 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:25.019 INFO:teuthology.orchestra.run.smithi163.stdout:Total 503 kB/s | 964 kB 00:01 2024-03-25T18:34:25.019 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:25.042 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:25.042 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:25.118 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:25.119 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:25.259 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:25.300 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:25.413 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repository Size 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:34:25.414 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout:Install 2 Packages 2024-03-25T18:34:25.415 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:25.416 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 332 k 2024-03-25T18:34:25.416 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.5 M 2024-03-25T18:34:25.416 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:25.442 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T18:34:25.508 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:25.597 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 378 kB/s | 69 kB 00:00 2024-03-25T18:34:26.224 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:34:26.225 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:34:26.514 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 240 kB/s | 264 kB 00:01 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:26.515 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:26.516 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:26.516 INFO:teuthology.orchestra.run.smithi130.stdout:Total 302 kB/s | 332 kB 00:01 2024-03-25T18:34:26.516 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:26.526 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:26.526 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:26.568 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:26.568 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:26.720 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:26.767 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-volume 2024-03-25T18:34:26.903 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:34:26.941 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:26.957 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:26.957 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:26.957 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:34:26.957 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:27.344 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:00 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:27.496 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T18:34:27.497 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:27.498 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:27.498 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:27.498 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-25T18:34:27.498 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:27.499 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 332 k 2024-03-25T18:34:27.499 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.5 M 2024-03-25T18:34:27.499 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:27.664 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 415 kB/s | 69 kB 00:00 2024-03-25T18:34:27.717 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:34:28.035 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:28.036 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:28.371 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados-devel 2024-03-25T18:34:28.372 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 302 kB/s | 264 kB 00:00 2024-03-25T18:34:28.374 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:28.374 INFO:teuthology.orchestra.run.smithi163.stdout:Total 379 kB/s | 332 kB 00:00 2024-03-25T18:34:28.374 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:28.384 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:28.384 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:28.427 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:28.428 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:28.581 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:28.774 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T18:34:28.817 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T18:34:28.833 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:28.833 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T18:34:28.833 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:28.951 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:29.101 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:29.101 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:29.101 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:34:29.102 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:29.103 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 127 k 2024-03-25T18:34:29.103 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 456 k 2024-03-25T18:34:29.103 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:29.621 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T18:34:29.786 INFO:teuthology.orchestra.run.smithi130.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 185 kB/s | 127 kB 00:00 2024-03-25T18:34:29.786 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:29.787 INFO:teuthology.orchestra.run.smithi130.stdout:Total 185 kB/s | 127 kB 00:00 2024-03-25T18:34:29.787 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:29.792 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:29.792 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:29.848 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:29.849 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T18:34:29.954 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:29.955 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:29.997 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:30.100 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:30.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel 2024-03-25T18:34:30.550 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:30.830 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:30.873 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:03 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T18:34:31.022 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 127 k 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 456 k 2024-03-25T18:34:31.023 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:31.053 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs2 2024-03-25T18:34:31.623 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:31.721 INFO:teuthology.orchestra.run.smithi130.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:31.778 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:31.779 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:31.779 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:31.841 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs-devel 2024-03-25T18:34:31.847 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 154 kB/s | 127 kB 00:00 2024-03-25T18:34:31.848 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:31.848 INFO:teuthology.orchestra.run.smithi163.stdout:Total 153 kB/s | 127 kB 00:00 2024-03-25T18:34:31.848 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:31.853 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:31.854 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:31.910 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:31.911 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:32.058 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:32.153 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:32.406 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:32.559 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:32.560 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:32.560 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:34:32.560 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 31 k 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 140 k 2024-03-25T18:34:32.561 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:32.625 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:32.929 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:32.929 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:32.929 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:32.929 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:32.929 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:32.930 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:33.105 INFO:teuthology.orchestra.run.smithi130.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 57 kB/s | 31 kB 00:00 2024-03-25T18:34:33.105 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:33.106 INFO:teuthology.orchestra.run.smithi130.stdout:Total 57 kB/s | 31 kB 00:00 2024-03-25T18:34:33.106 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:33.114 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:33.114 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:33.135 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:33.136 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:33.160 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2 2024-03-25T18:34:33.267 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:33.396 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:33.732 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:33.753 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:33.832 INFO:teuthology.orchestra.run.smithi163.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:33.886 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:33.887 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:33.888 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:33.963 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs-devel 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:34.032 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:34.189 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados2 2024-03-25T18:34:34.532 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:34.683 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:34.684 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 31 k 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 140 k 2024-03-25T18:34:34.685 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:34.753 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:34.850 INFO:teuthology.orchestra.run.smithi130.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:34.904 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:34.905 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:34.905 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:34.960 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librbd1 2024-03-25T18:34:35.118 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 72 kB/s | 31 kB 00:00 2024-03-25T18:34:35.119 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:35.119 INFO:teuthology.orchestra.run.smithi163.stdout:Total 72 kB/s | 31 kB 00:00 2024-03-25T18:34:35.119 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:35.123 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:35.123 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:35.154 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:35.154 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:35.283 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:35.375 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:35.523 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:35.621 INFO:teuthology.orchestra.run.smithi130.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:35.673 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:35.674 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:35.674 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:35.721 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rados 2024-03-25T18:34:35.765 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:36.055 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:36.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2 2024-03-25T18:34:36.297 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:36.402 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:36.457 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:36.458 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:36.458 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:36.520 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rgw 2024-03-25T18:34:36.785 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:36.883 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:36.935 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:36.936 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:36.936 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:36.989 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1 2024-03-25T18:34:37.105 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:37.205 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:37.259 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:37.261 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:37.261 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:37.307 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-cephfs 2024-03-25T18:34:37.553 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:37.649 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:37.701 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:37.702 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:37.702 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:37.753 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados 2024-03-25T18:34:37.875 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:37.971 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:38.022 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:38.024 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:38.024 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:38.075 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rbd 2024-03-25T18:34:38.315 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:38.411 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:38.463 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:38.464 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:38.464 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:38.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw 2024-03-25T18:34:38.656 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:38.753 INFO:teuthology.orchestra.run.smithi130.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:38.805 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:38.806 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T18:34:38.806 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:38.854 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-fuse 2024-03-25T18:34:39.073 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:39.170 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:39.222 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:39.223 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:39.223 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:39.281 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs 2024-03-25T18:34:39.417 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:39.566 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:39.566 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:39.566 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 85 k 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 231 k 2024-03-25T18:34:39.567 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:39.849 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:39.946 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:39.997 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:39.999 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:39.999 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:40.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd 2024-03-25T18:34:40.242 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 126 kB/s | 85 kB 00:00 2024-03-25T18:34:40.242 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:40.242 INFO:teuthology.orchestra.run.smithi130.stdout:Total 126 kB/s | 85 kB 00:00 2024-03-25T18:34:40.242 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:40.250 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:40.250 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:40.306 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:40.306 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:40.439 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:40.548 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:40.640 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:40.742 INFO:teuthology.orchestra.run.smithi163.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:34:40.796 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:40.797 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T18:34:40.797 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:40.854 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse 2024-03-25T18:34:40.954 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:41.289 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:41.440 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:41.518 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-mirror 2024-03-25T18:34:41.588 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:41.589 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:34:41.590 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:41.590 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 85 k 2024-03-25T18:34:41.590 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 231 k 2024-03-25T18:34:41.590 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:42.092 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:42.197 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 140 kB/s | 85 kB 00:00 2024-03-25T18:34:42.198 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:42.198 INFO:teuthology.orchestra.run.smithi163.stdout:Total 140 kB/s | 85 kB 00:00 2024-03-25T18:34:42.198 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:42.205 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:42.205 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:42.244 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:42.245 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:42.246 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:34:42.246 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:42.246 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 3.1 M 2024-03-25T18:34:42.246 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 13 M 2024-03-25T18:34:42.246 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:42.261 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:42.262 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:42.398 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:42.495 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:42.938 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:43.254 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:43.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror 2024-03-25T18:34:44.049 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:44.200 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:34:44.201 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:44.202 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 3.1 M 2024-03-25T18:34:44.202 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 13 M 2024-03-25T18:34:44.202 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:51.251 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 351 kB/s | 3.1 MB 00:09 2024-03-25T18:34:51.252 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:51.253 INFO:teuthology.orchestra.run.smithi130.stdout:Total 351 kB/s | 3.1 MB 00:09 2024-03-25T18:34:51.253 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:51.264 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:51.264 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:51.340 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:51.341 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:51.773 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:51.843 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:34:51.868 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:53.011 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:53.011 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:53.011 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:53.011 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:53.012 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:53.012 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:53.240 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-nbd 2024-03-25T18:34:53.820 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:53.967 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout: Package Architecture Version Repository Size 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:53.968 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 171 k 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 494 k 2024-03-25T18:34:53.969 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:54.758 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 299 kB/s | 3.1 MB 00:10 2024-03-25T18:34:54.759 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:54.760 INFO:teuthology.orchestra.run.smithi163.stdout:Total 299 kB/s | 3.1 MB 00:10 2024-03-25T18:34:54.760 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:54.771 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:54.771 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:54.795 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 208 kB/s | 171 kB 00:00 2024-03-25T18:34:54.796 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:54.796 INFO:teuthology.orchestra.run.smithi130.stdout:Total 207 kB/s | 171 kB 00:00 2024-03-25T18:34:54.796 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:34:54.805 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:34:54.805 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:34:54.847 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:54.847 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:54.862 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:34:54.863 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:34:55.013 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:34:55.118 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:55.273 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:55.334 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T18:34:55.359 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:55.566 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:55.851 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:55.851 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:55.852 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:34:55.852 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:55.852 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:55.852 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:34:56.055 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:34:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:56.482 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:56.482 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:56.483 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:56.623 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:34:56.712 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd 2024-03-25T18:34:56.776 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:34:56.776 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:56.776 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:34:56.776 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Installing dependencies: 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout:Install 2 Packages 2024-03-25T18:34:56.777 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:34:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 91 M 2024-03-25T18:34:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 498 M 2024-03-25T18:34:56.778 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:34:57.288 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:34:57.439 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:34:57.439 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:57.439 INFO:teuthology.orchestra.run.smithi163.stdout: Package Architecture Version Repository Size 2024-03-25T18:34:57.439 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:57.439 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 171 k 2024-03-25T18:34:57.440 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 494 k 2024-03-25T18:34:57.441 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:34:58.271 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 207 kB/s | 171 kB 00:00 2024-03-25T18:34:58.272 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:34:58.272 INFO:teuthology.orchestra.run.smithi163.stdout:Total 206 kB/s | 171 kB 00:00 2024-03-25T18:34:58.272 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:34:58.280 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:34:58.280 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:34:58.338 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:34:58.338 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:34:58.493 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:34:58.595 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:59.041 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:34:59.325 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:34:59.530 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:35:00.105 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T18:35:00.260 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:Install 2 Packages 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 91 M 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 498 M 2024-03-25T18:35:00.261 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:35:28.997 INFO:teuthology.orchestra.run.smithi163.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 800 kB/s | 22 MB 00:28 2024-03-25T18:35:56.490 INFO:teuthology.orchestra.run.smithi130.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 385 kB/s | 22 MB 00:59 2024-03-25T18:37:13.532 INFO:teuthology.orchestra.run.smithi163.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 524 kB/s | 68 MB 02:13 2024-03-25T18:37:13.532 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:13.533 INFO:teuthology.orchestra.run.smithi163.stdout:Total 696 kB/s | 91 MB 02:13 2024-03-25T18:37:13.533 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:37:13.537 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:37:13.538 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:37:14.124 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:37:14.124 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:37:17.252 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:37:24.473 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:37:24.616 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:37:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:37:28.050 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:37:28.323 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:37:28.324 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:37:28.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:37:29.119 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:08:02 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:37:29.278 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:37:29.279 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 416 M 2024-03-25T18:37:29.280 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.7 G 2024-03-25T18:37:29.280 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:37:38.643 INFO:teuthology.orchestra.run.smithi130.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 431 kB/s | 68 MB 02:41 2024-03-25T18:37:38.644 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:37:38.644 INFO:teuthology.orchestra.run.smithi130.stdout:Total 573 kB/s | 91 MB 02:41 2024-03-25T18:37:38.644 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:37:38.648 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:37:38.648 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:37:39.233 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:37:39.234 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:37:42.324 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:37:49.500 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T18:37:49.609 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:37:52.926 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T18:37:52.926 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:37:53.189 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:37:53.399 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-common-debuginfo 2024-03-25T18:37:53.975 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:08:30 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:37:54.131 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:37:54.132 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:37:54.133 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 416 M 2024-03-25T18:37:54.133 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.7 G 2024-03-25T18:37:54.133 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:53:36.521 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 440 kB/s | 416 MB 16:07 2024-03-25T18:53:36.521 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:53:36.521 INFO:teuthology.orchestra.run.smithi163.stdout:Total 440 kB/s | 416 MB 16:07 2024-03-25T18:53:36.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:53:36.525 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:53:36.525 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:53:38.995 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:53:38.995 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:54:21.335 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:54:21.430 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:28.606 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:28.873 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:54:29.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:54:29.695 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:25:02 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:54:29.858 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:54:29.858 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================== 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 1.4 M 2024-03-25T18:54:29.859 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 6.4 M 2024-03-25T18:54:29.860 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:54:32.501 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 553 kB/s | 1.4 MB 00:02 2024-03-25T18:54:32.502 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:54:32.502 INFO:teuthology.orchestra.run.smithi163.stdout:Total 553 kB/s | 1.4 MB 00:02 2024-03-25T18:54:32.503 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T18:54:32.507 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T18:54:32.507 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T18:54:32.554 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T18:54:32.554 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T18:54:32.815 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T18:54:32.908 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:33.307 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:33.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:54:33.589 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:33.589 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T18:54:33.589 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:54:33.590 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:33.590 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T18:54:33.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:54:34.334 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:25:07 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T18:54:34.484 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T18:54:34.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T18:54:34.485 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 176 M 2024-03-25T18:54:34.486 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 753 M 2024-03-25T18:54:34.486 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T18:56:13.691 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 387 kB/s | 416 MB 18:19 2024-03-25T18:56:13.692 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:56:13.692 INFO:teuthology.orchestra.run.smithi130.stdout:Total 387 kB/s | 416 MB 18:19 2024-03-25T18:56:13.692 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:56:13.695 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:56:13.695 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:56:16.170 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:56:16.171 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:56:58.556 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:56:58.659 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:03.635 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:03.895 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:57:03.895 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:03.895 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:57:03.895 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:03.896 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:03.896 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:57:04.166 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T18:57:04.773 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:27:41 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:57:04.926 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:======================================================================================== 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:04.927 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 1.4 M 2024-03-25T18:57:04.928 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 6.4 M 2024-03-25T18:57:04.928 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T18:57:10.146 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 280 kB/s | 1.4 MB 00:05 2024-03-25T18:57:10.146 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:57:10.146 INFO:teuthology.orchestra.run.smithi130.stdout:Total 280 kB/s | 1.4 MB 00:05 2024-03-25T18:57:10.147 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T18:57:10.151 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T18:57:10.151 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T18:57:10.196 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T18:57:10.196 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T18:57:10.456 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T18:57:10.552 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:10.949 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:11.233 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T18:57:11.427 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T18:57:11.981 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:27:48 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T18:57:12.130 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 176 M 2024-03-25T18:57:12.131 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 753 M 2024-03-25T18:57:12.133 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:00:18.088 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 524 kB/s | 176 MB 05:43 2024-03-25T19:00:18.089 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:00:18.089 INFO:teuthology.orchestra.run.smithi163.stdout:Total 524 kB/s | 176 MB 05:43 2024-03-25T19:00:18.089 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:00:18.093 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:00:18.093 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:00:19.153 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:00:19.153 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:00:37.402 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:00:37.551 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:42.294 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:42.551 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:00:42.551 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:00:42.551 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:00:42.551 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:00:42.551 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:00:42.552 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:00:42.764 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:00:43.334 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:31:16 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:00:43.483 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:00:43.484 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:00:43.485 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:00:43.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:00:43.485 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 672 M 2024-03-25T19:00:43.485 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 3.1 G 2024-03-25T19:00:43.485 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:04:46.871 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 396 kB/s | 176 MB 07:34 2024-03-25T19:04:46.872 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:04:46.872 INFO:teuthology.orchestra.run.smithi130.stdout:Total 396 kB/s | 176 MB 07:34 2024-03-25T19:04:46.872 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:04:46.875 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:04:46.875 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:04:47.933 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:04:47.934 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:05:06.146 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:05:06.317 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:10.895 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:11.146 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T19:05:11.146 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:05:11.146 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:05:11.147 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:05:11.147 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:05:11.147 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:05:11.348 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-test-debuginfo 2024-03-25T19:05:11.917 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 0:35:48 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:05:12.066 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 672 M 2024-03-25T19:05:12.067 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 3.1 G 2024-03-25T19:05:12.068 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:24:05.842 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 491 kB/s | 672 MB 23:22 2024-03-25T19:24:05.843 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:24:05.843 INFO:teuthology.orchestra.run.smithi163.stdout:Total 491 kB/s | 672 MB 23:22 2024-03-25T19:24:05.843 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:24:05.847 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:24:05.847 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:24:09.814 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:24:09.814 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:25:20.962 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:25:21.058 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:30.099 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:25:30.366 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:25:30.673 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:25:31.242 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:56:04 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:25:31.338 INFO:teuthology.orchestra.run.smithi163.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:25:31.390 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:25:31.391 INFO:teuthology.orchestra.run.smithi163.stdout:Nothing to do. 2024-03-25T19:25:31.392 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:25:31.463 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:25:32.022 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:56:05 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:25:32.169 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:25:32.170 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 20 M 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 103 M 2024-03-25T19:25:32.171 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:26:11.199 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 517 kB/s | 20 MB 00:39 2024-03-25T19:26:11.199 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:26:11.200 INFO:teuthology.orchestra.run.smithi163.stdout:Total 517 kB/s | 20 MB 00:39 2024-03-25T19:26:11.200 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:26:11.204 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:26:11.204 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:26:11.354 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:26:11.354 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:26:13.588 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:26:13.696 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:14.598 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:26:14.875 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:26:14.875 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:26:14.876 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:26:14.876 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:26:14.876 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:26:15.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:26:15.663 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:56:48 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:26:15.812 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:26:15.813 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 48 M 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 173 M 2024-03-25T19:26:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:27:10.919 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 887 kB/s | 48 MB 00:55 2024-03-25T19:27:10.920 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:27:10.920 INFO:teuthology.orchestra.run.smithi163.stdout:Total 887 kB/s | 48 MB 00:55 2024-03-25T19:27:10.920 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:27:10.923 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:27:10.923 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:27:11.235 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:27:11.236 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:27:15.978 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:27:16.100 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:17.521 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:17.781 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:27:17.781 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:27:17.781 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:27:17.781 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:27:17.782 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:27:17.782 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:27:17.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:27:18.537 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 0:57:51 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:27:18.688 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:27:18.689 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:27:18.689 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:27:18.689 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 99 M 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 449 M 2024-03-25T19:27:18.690 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 873 kB/s | 99 MB 01:55 2024-03-25T19:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:29:14.198 INFO:teuthology.orchestra.run.smithi163.stdout:Total 873 kB/s | 99 MB 01:55 2024-03-25T19:29:14.199 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:29:14.201 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:29:14.811 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:29:14.811 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:29:25.112 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:29:25.203 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:28.165 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:28.427 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:29:28.427 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:29:28.427 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:29:28.427 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:29:28.427 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:29:28.428 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:29:28.656 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:29:29.226 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:00:02 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:29:29.377 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:29:29.378 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 287 M 2024-03-25T19:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.2 G 2024-03-25T19:29:29.379 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:30:15.040 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 458 kB/s | 672 MB 25:02 2024-03-25T19:30:15.040 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:30:15.041 INFO:teuthology.orchestra.run.smithi130.stdout:Total 458 kB/s | 672 MB 25:02 2024-03-25T19:30:15.041 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:30:15.046 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:30:15.046 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:30:19.007 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:30:19.007 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:31:29.712 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:31:30.325 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:38.606 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:38.859 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:31:38.860 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:38.860 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:31:38.860 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:38.860 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:38.860 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:31:39.167 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-base-debuginfo 2024-03-25T19:31:39.748 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:02:16 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:31:39.853 INFO:teuthology.orchestra.run.smithi130.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T19:31:39.908 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:31:39.909 INFO:teuthology.orchestra.run.smithi130.stdout:Nothing to do. 2024-03-25T19:31:39.909 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:31:39.974 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T19:31:40.562 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:02:17 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:40.720 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 20 M 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 103 M 2024-03-25T19:31:40.721 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:31:56.116 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 20 MB 00:15 2024-03-25T19:31:56.117 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:31:56.117 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.3 MB/s | 20 MB 00:15 2024-03-25T19:31:56.118 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:31:56.121 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:31:56.122 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:31:56.271 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:31:56.272 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:31:58.492 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:31:58.588 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:59.525 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:31:59.790 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:31:59.988 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T19:32:00.548 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:02:37 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:32:00.699 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:32:00.700 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:00.700 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:32:00.700 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 48 M 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 173 M 2024-03-25T19:32:00.701 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:32:30.998 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.6 MB/s | 48 MB 00:30 2024-03-25T19:32:30.998 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:32:30.999 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.6 MB/s | 48 MB 00:30 2024-03-25T19:32:30.999 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:32:31.003 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:32:31.003 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:32:31.319 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:32:31.319 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:32:36.057 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:32:36.154 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:37.490 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:37.753 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:32:37.951 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T19:32:38.515 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:03:15 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:32:38.664 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:32:38.665 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:32:38.666 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 99 M 2024-03-25T19:32:38.666 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 449 M 2024-03-25T19:32:38.666 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:34:12.813 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 99 MB 01:34 2024-03-25T19:34:12.813 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:12.814 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.0 MB/s | 99 MB 01:34 2024-03-25T19:34:12.814 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:34:12.817 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:34:12.817 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:34:13.424 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:34:13.424 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:34:23.704 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:34:23.800 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:26.702 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:34:26.958 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:34:27.165 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T19:34:27.732 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:05:04 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:34:27.880 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:34:27.881 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:34:27.882 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:34:27.882 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 287 M 2024-03-25T19:34:27.882 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.2 G 2024-03-25T19:34:27.882 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:34:52.811 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 908 kB/s | 287 MB 05:23 2024-03-25T19:34:52.811 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:34:52.812 INFO:teuthology.orchestra.run.smithi163.stdout:Total 908 kB/s | 287 MB 05:23 2024-03-25T19:34:52.812 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:34:52.815 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:34:52.815 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:34:54.522 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:34:54.522 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:35:23.890 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:35:23.997 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:31.541 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:31.819 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:35:31.819 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:31.819 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:35:31.819 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:31.820 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:31.820 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:35:32.047 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:35:32.625 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:06:05 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:35:32.785 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:35:32.786 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:32.787 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:35:32.787 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:32.787 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 5.9 M 2024-03-25T19:35:32.787 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 28 M 2024-03-25T19:35:32.787 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:35:36.893 INFO:teuthology.orchestra.run.smithi163.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.4 MB/s | 5.9 MB 00:04 2024-03-25T19:35:36.894 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:36.895 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.4 MB/s | 5.9 MB 00:04 2024-03-25T19:35:36.895 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:35:36.899 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:35:36.899 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:35:36.971 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:35:36.972 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:35:37.704 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:35:37.817 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:38.369 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:38.638 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:35:38.860 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:35:39.419 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:06:12 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:35:39.570 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:39.571 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:35:39.572 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:39.572 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 109 k 2024-03-25T19:35:39.572 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 440 k 2024-03-25T19:35:39.572 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:35:40.280 INFO:teuthology.orchestra.run.smithi163.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 153 kB/s | 109 kB 00:00 2024-03-25T19:35:40.281 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:40.281 INFO:teuthology.orchestra.run.smithi163.stdout:Total 153 kB/s | 109 kB 00:00 2024-03-25T19:35:40.281 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:35:40.284 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:35:40.284 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:35:40.318 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:35:40.318 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:35:40.436 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:35:40.536 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:40.946 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:41.202 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:35:41.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:35:41.962 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:06:14 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:35:42.110 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:35:42.111 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:42.111 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:35:42.111 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:42.111 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 5.4 M 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 25 M 2024-03-25T19:35:42.112 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:35:51.732 INFO:teuthology.orchestra.run.smithi163.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 570 kB/s | 5.4 MB 00:09 2024-03-25T19:35:51.733 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:35:51.733 INFO:teuthology.orchestra.run.smithi163.stdout:Total 570 kB/s | 5.4 MB 00:09 2024-03-25T19:35:51.733 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:35:51.737 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:35:51.737 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:35:51.806 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:35:51.807 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:35:52.476 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:35:52.589 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:35:53.131 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:35:53.393 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:35:53.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:53.394 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:35:53.394 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:35:53.394 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:53.394 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:35:53.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librados2-debuginfo 2024-03-25T19:35:54.189 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:06:27 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:35:54.340 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 44 M 2024-03-25T19:35:54.341 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 253 M 2024-03-25T19:35:54.342 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:36:38.854 INFO:teuthology.orchestra.run.smithi163.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 1.0 MB/s | 44 MB 00:44 2024-03-25T19:36:38.855 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:36:38.855 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.0 MB/s | 44 MB 00:44 2024-03-25T19:36:38.855 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:36:38.859 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:36:38.859 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:36:39.159 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:36:39.159 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:36:44.218 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:36:44.319 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:46.147 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:36:46.428 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:36:46.628 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:36:47.193 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:07:20 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:36:47.342 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:36:47.343 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:36:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:36:47.343 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:36:47.343 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:36:47.343 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 52 M 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 308 M 2024-03-25T19:36:47.344 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:20.719 INFO:teuthology.orchestra.run.smithi163.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.6 MB/s | 52 MB 00:33 2024-03-25T19:37:20.720 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:20.721 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.6 MB/s | 52 MB 00:33 2024-03-25T19:37:20.721 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:20.725 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:20.725 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:21.071 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:21.071 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:27.058 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:27.160 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:29.370 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:29.635 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:37:29.635 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:29.635 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:29.635 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:29.635 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:29.636 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:29.852 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:37:30.413 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:03 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:30.562 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 615 k 2024-03-25T19:37:30.563 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.1 M 2024-03-25T19:37:30.564 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:31.406 INFO:teuthology.orchestra.run.smithi163.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 730 kB/s | 615 kB 00:00 2024-03-25T19:37:31.409 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:31.409 INFO:teuthology.orchestra.run.smithi163.stdout:Total 729 kB/s | 615 kB 00:00 2024-03-25T19:37:31.410 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:31.411 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:31.411 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:31.452 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:31.452 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:31.621 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:31.738 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:32.169 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:32.435 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:37:32.435 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:32.436 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:32.436 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:32.436 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:32.436 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:32.649 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:37:33.211 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:06 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:33.362 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:37:33.364 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 771 k 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.7 M 2024-03-25T19:37:33.365 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:34.443 INFO:teuthology.orchestra.run.smithi163.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 715 kB/s | 771 kB 00:01 2024-03-25T19:37:34.443 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:34.444 INFO:teuthology.orchestra.run.smithi163.stdout:Total 714 kB/s | 771 kB 00:01 2024-03-25T19:37:34.444 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:34.448 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:34.448 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:34.489 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:34.490 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:34.668 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:34.766 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:35.150 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:35.408 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:37:35.409 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:35.409 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:35.409 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:35.409 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:35.409 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:35.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:37:36.191 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:09 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:36.341 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:36.341 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:36.341 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:36.341 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 1.1 M 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 4.3 M 2024-03-25T19:37:36.342 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:37.855 INFO:teuthology.orchestra.run.smithi163.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 761 kB/s | 1.1 MB 00:01 2024-03-25T19:37:37.855 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:37.856 INFO:teuthology.orchestra.run.smithi163.stdout:Total 760 kB/s | 1.1 MB 00:01 2024-03-25T19:37:37.856 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:37.860 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:37.860 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:37.904 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:37.905 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:38.125 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:38.237 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:38.685 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:38.951 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:39.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:37:39.735 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:12 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:39.884 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:39.885 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:39.886 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 337 k 2024-03-25T19:37:39.886 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 1.0 M 2024-03-25T19:37:39.886 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:40.883 INFO:teuthology.orchestra.run.smithi163.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 338 kB/s | 337 kB 00:00 2024-03-25T19:37:40.883 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:40.884 INFO:teuthology.orchestra.run.smithi163.stdout:Total 338 kB/s | 337 kB 00:00 2024-03-25T19:37:40.884 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:40.888 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:40.888 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:40.927 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:40.928 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:41.068 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:41.168 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:41.562 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:41.824 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:37:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:41.825 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:41.825 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:41.825 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:42.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:37:42.610 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:15 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:42.759 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:42.760 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 509 k 2024-03-25T19:37:42.761 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 2.5 M 2024-03-25T19:37:42.761 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:37:45.130 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 215 kB/s | 509 kB 00:02 2024-03-25T19:37:45.130 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:37:45.131 INFO:teuthology.orchestra.run.smithi163.stdout:Total 215 kB/s | 509 kB 00:02 2024-03-25T19:37:45.131 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:37:45.135 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:37:45.135 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:37:45.176 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:37:45.177 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:37:45.340 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:37:45.446 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:45.838 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:46.098 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:37:46.098 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:46.098 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:37:46.099 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:37:46.099 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:46.099 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:37:46.298 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:37:46.868 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:08:19 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:37:47.018 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:37:47.019 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:37:47.020 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 52 M 2024-03-25T19:37:47.020 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 298 M 2024-03-25T19:37:47.020 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:38:24.444 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 1.4 MB/s | 52 MB 00:37 2024-03-25T19:38:24.445 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:24.445 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.4 MB/s | 52 MB 00:37 2024-03-25T19:38:24.445 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:38:24.449 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:38:24.449 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:38:24.786 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:38:24.786 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:38:30.716 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:38:31.026 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:33.151 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:38:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:33.424 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:38:33.424 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:33.425 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:33.425 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:38:33.627 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:38:34.194 INFO:teuthology.orchestra.run.smithi163.stdout:Last metadata expiration check: 1:09:07 ago on Mon 25 Mar 2024 06:29:27 PM UTC. 2024-03-25T19:38:34.345 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repo Size 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:================================================================================ 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout:Install 1 Package 2024-03-25T19:38:34.346 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:34.347 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 1.3 M 2024-03-25T19:38:34.347 INFO:teuthology.orchestra.run.smithi163.stdout:Installed size: 5.5 M 2024-03-25T19:38:34.347 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-03-25T19:38:35.412 INFO:teuthology.orchestra.run.smithi163.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:38:35.413 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:35.413 INFO:teuthology.orchestra.run.smithi163.stdout:Total 1.2 MB/s | 1.3 MB 00:01 2024-03-25T19:38:35.414 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-03-25T19:38:35.417 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-03-25T19:38:35.417 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-03-25T19:38:35.462 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-03-25T19:38:35.462 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-03-25T19:38:35.717 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-03-25T19:38:35.811 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:36.208 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:36.472 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:38:36.472 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:36.472 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-03-25T19:38:36.473 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:38:36.473 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:38:36.473 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-03-25T19:38:36.712 DEBUG:teuthology.parallel:result is None 2024-03-25T19:38:54.519 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 287 MB 04:26 2024-03-25T19:38:54.520 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:38:54.520 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.1 MB/s | 287 MB 04:26 2024-03-25T19:38:54.520 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:38:54.523 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:38:54.523 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:38:56.231 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:38:56.232 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:39:25.499 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:39:25.605 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:32.995 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:33.276 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:39:33.510 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T19:39:34.077 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:11 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T19:39:34.227 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 5.9 M 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 28 M 2024-03-25T19:39:34.228 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:39:35.218 INFO:teuthology.orchestra.run.smithi130.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 6.0 MB/s | 5.9 MB 00:00 2024-03-25T19:39:35.218 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:35.219 INFO:teuthology.orchestra.run.smithi130.stdout:Total 6.0 MB/s | 5.9 MB 00:00 2024-03-25T19:39:35.219 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:39:35.222 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:39:35.222 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:39:35.290 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:39:35.291 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:39:36.022 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:39:36.125 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:36.684 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:36.948 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:39:37.199 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados-devel-debuginfo 2024-03-25T19:39:37.762 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:14 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:39:37.914 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 109 k 2024-03-25T19:39:37.915 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 440 k 2024-03-25T19:39:37.916 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:39:38.129 INFO:teuthology.orchestra.run.smithi130.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 509 kB/s | 109 kB 00:00 2024-03-25T19:39:38.130 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:38.130 INFO:teuthology.orchestra.run.smithi130.stdout:Total 506 kB/s | 109 kB 00:00 2024-03-25T19:39:38.130 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:39:38.133 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:39:38.133 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:39:38.168 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:39:38.168 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:39:38.286 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:39:38.395 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:38.785 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:39.058 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:39:39.059 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:39.059 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:39:39.059 INFO:teuthology.orchestra.run.smithi130.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:39.059 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:39.059 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:39:39.279 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T19:39:39.843 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:16 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:39:39.992 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:39:39.993 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:39.993 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:39:39.993 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:39.993 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 5.4 M 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 25 M 2024-03-25T19:39:39.994 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:39:40.674 INFO:teuthology.orchestra.run.smithi130.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 7.9 MB/s | 5.4 MB 00:00 2024-03-25T19:39:40.674 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:40.674 INFO:teuthology.orchestra.run.smithi130.stdout:Total 7.9 MB/s | 5.4 MB 00:00 2024-03-25T19:39:40.675 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:39:40.678 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:39:40.679 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:39:40.745 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:39:40.745 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:39:41.406 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:39:41.513 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:42.008 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:42.290 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:39:42.291 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:42.291 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:39:42.291 INFO:teuthology.orchestra.run.smithi130.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:42.291 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:42.291 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:39:42.550 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librados2-debuginfo 2024-03-25T19:39:43.113 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:20 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:39:43.265 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:39:43.265 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:43.265 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 44 M 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 253 M 2024-03-25T19:39:43.266 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:39:47.176 INFO:teuthology.orchestra.run.smithi130.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 11 MB/s | 44 MB 00:03 2024-03-25T19:39:47.176 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:47.177 INFO:teuthology.orchestra.run.smithi130.stdout:Total 11 MB/s | 44 MB 00:03 2024-03-25T19:39:47.177 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:39:47.181 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:39:47.181 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:39:47.481 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:39:47.481 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:39:52.545 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:39:52.633 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:54.455 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:54.726 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:39:54.968 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install librbd1-debuginfo 2024-03-25T19:39:55.535 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:32 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:39:55.685 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:39:55.685 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:55.685 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:39:55.685 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:39:55.686 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 52 M 2024-03-25T19:39:55.687 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 308 M 2024-03-25T19:39:55.687 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:39:59.728 INFO:teuthology.orchestra.run.smithi130.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 13 MB/s | 52 MB 00:04 2024-03-25T19:39:59.728 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:39:59.729 INFO:teuthology.orchestra.run.smithi130.stdout:Total 13 MB/s | 52 MB 00:04 2024-03-25T19:39:59.729 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:39:59.733 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:39:59.733 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:00.079 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:00.079 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:06.061 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:06.159 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:08.268 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:08.535 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:08.535 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:08.536 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:08.536 INFO:teuthology.orchestra.run.smithi130.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:08.536 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:08.536 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:08.758 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T19:40:09.319 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:46 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:09.468 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:09.469 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:09.469 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:09.469 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:09.469 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:09.469 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 615 k 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.1 M 2024-03-25T19:40:09.470 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:10.105 INFO:teuthology.orchestra.run.smithi130.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 968 kB/s | 615 kB 00:00 2024-03-25T19:40:10.106 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:10.106 INFO:teuthology.orchestra.run.smithi130.stdout:Total 966 kB/s | 615 kB 00:00 2024-03-25T19:40:10.106 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:10.110 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:10.110 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:10.149 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:10.150 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:10.315 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:10.429 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:10.826 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:11.088 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:11.311 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rados-debuginfo 2024-03-25T19:40:11.876 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:48 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:12.027 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:12.028 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:12.029 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 771 k 2024-03-25T19:40:12.029 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.7 M 2024-03-25T19:40:12.029 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:13.000 INFO:teuthology.orchestra.run.smithi130.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 794 kB/s | 771 kB 00:00 2024-03-25T19:40:13.001 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:13.001 INFO:teuthology.orchestra.run.smithi130.stdout:Total 793 kB/s | 771 kB 00:00 2024-03-25T19:40:13.001 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:13.005 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:13.005 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:13.044 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:13.044 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:13.223 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:13.324 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:13.693 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:13.952 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:14.200 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T19:40:14.763 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:51 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:14.913 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:14.913 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 1.1 M 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 4.3 M 2024-03-25T19:40:14.914 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:15.245 INFO:teuthology.orchestra.run.smithi130.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 3.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:15.245 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:15.246 INFO:teuthology.orchestra.run.smithi130.stdout:Total 3.4 MB/s | 1.1 MB 00:00 2024-03-25T19:40:15.246 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:15.249 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:15.249 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:15.289 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:15.289 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:15.510 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:15.619 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.007 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.271 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:16.271 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:16.271 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:16.271 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:16.272 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:16.272 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:16.521 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T19:40:17.086 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:54 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:17.237 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 337 k 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 1.0 M 2024-03-25T19:40:17.238 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:17.489 INFO:teuthology.orchestra.run.smithi130.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 1.3 MB/s | 337 kB 00:00 2024-03-25T19:40:17.490 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:17.490 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.3 MB/s | 337 kB 00:00 2024-03-25T19:40:17.490 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:17.493 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:17.494 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:17.528 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:17.529 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:17.671 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:17.781 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.132 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.391 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T19:40:18.391 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:18.391 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:18.391 INFO:teuthology.orchestra.run.smithi130.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:18.391 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:18.392 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:18.612 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T19:40:19.176 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:56 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:19.326 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:19.327 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 509 k 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 2.5 M 2024-03-25T19:40:19.328 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:19.717 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 1.3 MB/s | 509 kB 00:00 2024-03-25T19:40:19.718 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:19.719 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.3 MB/s | 509 kB 00:00 2024-03-25T19:40:19.719 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:19.722 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:19.722 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:19.758 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:19.759 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:19.917 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:20.034 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.404 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.688 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T19:40:20.689 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:20.689 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:20.689 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:20.689 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:20.689 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:20.918 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T19:40:21.485 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:10:58 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:21.634 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 52 M 2024-03-25T19:40:21.635 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 298 M 2024-03-25T19:40:21.636 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:26.711 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 10 MB/s | 52 MB 00:05 2024-03-25T19:40:26.712 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:26.712 INFO:teuthology.orchestra.run.smithi130.stdout:Total 10 MB/s | 52 MB 00:05 2024-03-25T19:40:26.713 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:26.717 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:26.717 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:27.051 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:27.052 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:32.946 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:33.269 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:35.327 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:35.589 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:35.807 DEBUG:teuthology.orchestra.run.smithi130:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T19:40:36.370 INFO:teuthology.orchestra.run.smithi130.stdout:Last metadata expiration check: 1:11:13 ago on Mon 25 Mar 2024 06:29:23 PM UTC. 2024-03-25T19:40:36.519 INFO:teuthology.orchestra.run.smithi130.stdout:Dependencies resolved. 2024-03-25T19:40:36.520 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:36.520 INFO:teuthology.orchestra.run.smithi130.stdout: Package Arch Version Repo Size 2024-03-25T19:40:36.520 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:36.520 INFO:teuthology.orchestra.run.smithi130.stdout:Installing: 2024-03-25T19:40:36.520 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction Summary 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:================================================================================ 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Install 1 Package 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Total download size: 1.3 M 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Installed size: 5.5 M 2024-03-25T19:40:36.521 INFO:teuthology.orchestra.run.smithi130.stdout:Downloading Packages: 2024-03-25T19:40:37.688 INFO:teuthology.orchestra.run.smithi130.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:40:37.689 INFO:teuthology.orchestra.run.smithi130.stdout:-------------------------------------------------------------------------------- 2024-03-25T19:40:37.689 INFO:teuthology.orchestra.run.smithi130.stdout:Total 1.1 MB/s | 1.3 MB 00:01 2024-03-25T19:40:37.689 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction check 2024-03-25T19:40:37.693 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction check succeeded. 2024-03-25T19:40:37.693 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction test 2024-03-25T19:40:37.739 INFO:teuthology.orchestra.run.smithi130.stdout:Transaction test succeeded. 2024-03-25T19:40:37.740 INFO:teuthology.orchestra.run.smithi130.stdout:Running transaction 2024-03-25T19:40:37.978 INFO:teuthology.orchestra.run.smithi130.stdout: Preparing : 1/1 2024-03-25T19:40:38.069 INFO:teuthology.orchestra.run.smithi130.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.485 INFO:teuthology.orchestra.run.smithi130.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout:Installed: 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:38.749 INFO:teuthology.orchestra.run.smithi130.stdout:Complete! 2024-03-25T19:40:38.982 DEBUG:teuthology.parallel:result is None 2024-03-25T19:40:38.983 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:39.125 DEBUG:teuthology.orchestra.run.smithi130:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:39.146 INFO:teuthology.orchestra.run.smithi130.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:39.146 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:39.146 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:39.148 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T19:40:39.286 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T19:40:39.311 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:39.311 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T19:40:39.311 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T19:40:39.312 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T19:40:39.313 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:39.313 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:39.340 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:39.340 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T19:40:39.382 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T19:40:39.383 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:39.383 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:39.410 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:39.476 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:39.477 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T19:40:39.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T19:40:39.577 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T19:40:39.577 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:39.577 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:39.603 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:39.670 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:39.670 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T19:40:39.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T19:40:39.769 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T19:40:39.769 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:39.769 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:39.796 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:39.863 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:39.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T19:40:39.891 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T19:40:39.959 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T19:40:40.037 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T19:40:40.038 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:40.040 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2024-03-25T19:40:40.068 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T19:40:40.068 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:40.108 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T19:40:40.135 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T19:40:40.135 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:40.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T19:40:40.212 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T19:40:40.212 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T19:40:40.212 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621322', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi130.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC151G++LGPgESxw1yLJIi18NohTBabOkdI3F3RowAdjaQ+fN969rAq2YN8RW/M5KRLIbbjm23W8VU5H6WnoUZA=', 'smithi163.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoqCjQcNIhRg3dFSzdt0NrGct9OOSowWbTVZL//H+1UrhCpY21F4j7tYptMsoDnwA2eSnEepdYgb6Ac2LGNpcQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T19:40:40.213 DEBUG:teuthology.orchestra.run.smithi130:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:40.241 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T19:40:40.257 DEBUG:teuthology.orchestra.run.smithi130:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:40.297 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T19:40:40.324 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:40.325 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:40.376 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:40.376 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-25 18:34:52.131782446 +0000 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-25 18:28:29.420400963 +0000 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-25 18:28:29.420400963 +0000 2024-03-25T19:40:40.431 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-25 18:28:29.420400963 +0000 2024-03-25T19:40:40.431 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:40.496 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-25T19:40:40.496 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-25T19:40:40.496 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000261014 s, 2.0 MB/s 2024-03-25T19:40:40.498 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:40.554 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-25 18:34:52.131782446 +0000 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-25 18:28:29.777394391 +0000 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-25 18:28:29.777394391 +0000 2024-03-25T19:40:40.609 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-25 18:28:29.777394391 +0000 2024-03-25T19:40:40.609 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:40.675 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-25T19:40:40.675 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-25T19:40:40.675 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000186982 s, 2.7 MB/s 2024-03-25T19:40:40.676 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:40.732 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:40.786 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:40.786 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:40.786 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-25 18:34:52.132782428 +0000 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-25 18:28:30.121388058 +0000 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-25 18:28:30.121388058 +0000 2024-03-25T19:40:40.787 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-25 18:28:30.121388058 +0000 2024-03-25T19:40:40.787 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:40.851 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-25T19:40:40.851 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-25T19:40:40.851 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000233077 s, 2.2 MB/s 2024-03-25T19:40:40.853 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:40.909 DEBUG:teuthology.orchestra.run.smithi130:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Access: 2024-03-25 18:34:52.132782428 +0000 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Modify: 2024-03-25 18:28:30.460381817 +0000 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout:Change: 2024-03-25 18:28:30.460381817 +0000 2024-03-25T19:40:40.964 INFO:teuthology.orchestra.run.smithi130.stdout: Birth: 2024-03-25 18:28:30.460381817 +0000 2024-03-25T19:40:40.964 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:41.029 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records in 2024-03-25T19:40:41.029 INFO:teuthology.orchestra.run.smithi130.stderr:1+0 records out 2024-03-25T19:40:41.029 INFO:teuthology.orchestra.run.smithi130.stderr:512 bytes copied, 0.000210501 s, 2.4 MB/s 2024-03-25T19:40:41.030 DEBUG:teuthology.orchestra.run.smithi130:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:41.085 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:41.086 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:41.086 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T19:40:41.104 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T19:40:41.104 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-25 18:34:55.615718262 +0000 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-25 18:28:29.818829753 +0000 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-25 18:28:29.818829753 +0000 2024-03-25T19:40:41.161 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-25 18:28:29.818829753 +0000 2024-03-25T19:40:41.162 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T19:40:41.230 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-25T19:40:41.230 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-25T19:40:41.230 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000317836 s, 1.6 MB/s 2024-03-25T19:40:41.231 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T19:40:41.289 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-25 18:34:55.616718238 +0000 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-25 18:28:30.157821493 +0000 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-25 18:28:30.157821493 +0000 2024-03-25T19:40:41.346 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-25 18:28:30.157821493 +0000 2024-03-25T19:40:41.346 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T19:40:41.414 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-25T19:40:41.466 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-25T19:40:41.466 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000217503 s, 2.4 MB/s 2024-03-25T19:40:41.467 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T19:40:41.526 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-03-25T19:40:41.582 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T19:40:41.582 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:41.582 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-25 18:34:55.616718238 +0000 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-25 18:28:30.496813234 +0000 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-25 18:28:30.496813234 +0000 2024-03-25T19:40:41.583 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-25 18:28:30.496813234 +0000 2024-03-25T19:40:41.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T19:40:41.650 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-25T19:40:41.650 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-25T19:40:41.650 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000225539 s, 2.3 MB/s 2024-03-25T19:40:41.651 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T19:40:41.707 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-03-25 18:34:55.616718238 +0000 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-03-25 18:28:30.853804536 +0000 2024-03-25T19:40:41.762 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-03-25 18:28:30.853804536 +0000 2024-03-25T19:40:41.763 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-03-25 18:28:30.853804536 +0000 2024-03-25T19:40:41.763 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T19:40:41.828 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-03-25T19:40:41.828 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-03-25T19:40:41.828 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000189337 s, 2.7 MB/s 2024-03-25T19:40:41.829 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T19:40:41.885 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:41.885 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi130.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:41.885 INFO:tasks.ceph:Generating config... 2024-03-25T19:40:41.886 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T19:40:41.886 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T19:40:41.886 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T19:40:41.886 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T19:40:41.886 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T19:40:41.886 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T19:40:41.887 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T19:40:41.887 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T19:40:41.887 INFO:tasks.ceph:[global] ms type = async 2024-03-25T19:40:41.887 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T19:40:41.887 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T19:40:41.888 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T19:40:41.888 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T19:40:41.889 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T19:40:41.890 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T19:40:41.890 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T19:40:41.939 INFO:teuthology.orchestra.run.smithi130.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T19:40:41.942 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T19:40:41.986 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:42.053 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.130:6789]'), ('mon.c', '[v1:172.21.15.130:6790]'), ('mon.b', '[v1:172.21.15.163:6789]')] 2024-03-25T19:40:42.054 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.130:6789],[v1:172.21.15.130:6790],[v1:172.21.15.163:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T19:40:42.054 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:42.055 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:42.108 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.130:6789]' --addv c '[v1:172.21.15.130:6790]' --addv b '[v1:172.21.15.163:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:42.192 INFO:teuthology.orchestra.run.smithi130.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:42.192 INFO:teuthology.orchestra.run.smithi130.stdout:monmaptool: generated fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:40:42.192 INFO:teuthology.orchestra.run.smithi130.stdout:setting min_mon_release = quincy 2024-03-25T19:40:42.192 INFO:teuthology.orchestra.run.smithi130.stdout:epoch 0 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:min_mon_release 17 (quincy) 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:election_strategy: 3 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:40:42.193 INFO:teuthology.orchestra.run.smithi130.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T19:40:42.194 DEBUG:teuthology.orchestra.run.smithi130:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T19:40:42.209 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 2da1777d-8d35-40d5-acd9-30de4511aae0... 2024-03-25T19:40:42.209 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:42.251 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout:[global] 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: chdir = "" 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: auth supported = cephx 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: filestore xattr use omap = true 2024-03-25T19:40:42.299 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: auth debug = true 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on old message = true 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: ms die on bug = true 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.300 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default size = 2 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd allow pg remap = true 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on too few osds = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.301 INFO:teuthology.orchestra.run.smithi130.stdout: osd default data pool replay window = 5 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon allow pool delete = true 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon cluster log file level = debug 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: debug asserts on shutdown = true 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon health detail to clog = false 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon host = "[v1:172.21.15.130:6789],[v1:172.21.15.130:6790],[v1:172.21.15.163:6789]" 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon client directed command retry = 5 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: mon election default strategy = 3 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: ms bind msgr2 = False 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: ms inject socket failures = 5000 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: ms type = async 2024-03-25T19:40:42.302 INFO:teuthology.orchestra.run.smithi130.stdout: osd heartbeat grace = 80 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: fsid = 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout:[osd] 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd journal size = 100 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd scrub max interval = 600 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd recover clone overlap = true 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug shutdown = true 2024-03-25T19:40:42.303 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug op order = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug trim objects = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd open classes on start = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug pg log writeout = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: osd map max advance = 10 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: journal zero on create = true 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:42.304 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bdev debug aio = true 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug misdirected ops = true 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bdev async discard = True 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bdev enable discard = True 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bluestore allocator = stupid 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bluestore block size = 96636764160 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bluestore fsck on mount = True 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: bluestore zero block detection = True 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: debug bluefs = 1 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: debug bluestore = 1 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: debug monc = 20 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2024-03-25T19:40:42.305 INFO:teuthology.orchestra.run.smithi130.stdout: debug osd = 20 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: debug rocksdb = 10 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd class default list = * 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd class load list = * 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd client watch timeout = 120 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd debug verify missing on start = True 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd fast shutdown = False 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd max markdown count = 1000 2024-03-25T19:40:42.306 INFO:teuthology.orchestra.run.smithi130.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: osd objectstore = bluestore 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: osd op queue = debug_random 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr] 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: debug mgr = 20 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.307 INFO:teuthology.orchestra.run.smithi130.stdout:[mon] 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: debug mon = 20 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: debug paxos = 20 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: debug auth = 20 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon data avail warn = 5 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd prime pg temp = true 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.308 INFO:teuthology.orchestra.run.smithi130.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: # 1m isn't quite enough 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon osd crush smoke test = False 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon scrub interval = 300 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: mon warn on pool no app = False 2024-03-25T19:40:42.309 INFO:teuthology.orchestra.run.smithi130.stdout: paxos service trim min = 10 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout:[client] 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: rgw cache enabled = true 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable ops log = true 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: rgw enable usage log = true 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: debug monc = 20 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: debug ms = 1 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: debug objecter = 20 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout: debug rados = 20 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.a] 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.c] 2024-03-25T19:40:42.310 INFO:teuthology.orchestra.run.smithi130.stdout:[mon.b] 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2024-03-25T19:40:42.311 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2024-03-25T19:40:42.312 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: mon_allow_pool_size_one = true 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2024-03-25T19:40:42.313 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "[v1:172.21.15.130:6789],[v1:172.21.15.130:6790],[v1:172.21.15.163:6789]" 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon client directed command retry = 5 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: mon election default strategy = 3 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: ms bind msgr2 = False 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: ms inject socket failures = 5000 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: ms type = async 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: osd heartbeat grace = 80 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:40:42.314 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock profile = high_recovery_ops 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2024-03-25T19:40:42.315 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug trim objects = true 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2024-03-25T19:40:42.316 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bdev async discard = True 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bdev enable discard = True 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore allocator = stupid 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore block size = 96636764160 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore fsck on mount = True 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: bluestore zero block detection = True 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluefs = 1 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug bluestore = 1 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug monc = 20 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: debug rocksdb = 10 2024-03-25T19:40:42.317 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd full ratio = 0.9 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd nearfull ratio = 0.8 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd blocked scrub grace period = 3600 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd class default list = * 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd class load list = * 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd client watch timeout = 120 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify cached snaps = True 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify missing on start = True 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd failsafe full ratio = 0.95 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd fast shutdown = False 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd max markdown count = 1000 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd mclock override recovery settings = True 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = bluestore 2024-03-25T19:40:42.318 INFO:teuthology.orchestra.run.smithi163.stdout: osd op queue = debug_random 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: osd op queue cut off = debug_random 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: osd shutdown pgref assert = True 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2024-03-25T19:40:42.319 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.320 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: # 1m isn't quite enough 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon_down_mkfs_grace = 2m 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_filestore_osds = false 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon min osdmap epochs = 50 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd crush smoke test = False 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune interval = 2 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune min = 15 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon osdmap full prune txsize = 2 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon scrub interval = 300 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on pool no app = False 2024-03-25T19:40:42.321 INFO:teuthology.orchestra.run.smithi163.stdout: paxos service trim min = 10 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: debug monc = 20 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: debug objecter = 20 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout: debug rados = 20 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2024-03-25T19:40:42.322 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2024-03-25T19:40:42.324 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T19:40:42.324 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T19:40:42.379 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T19:40:42.379 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:42.379 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T19:40:42.394 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:42.394 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T19:40:42.449 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:40:42.449 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:42.449 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:42.449 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:42.528 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:40:42.528 DEBUG:teuthology.orchestra.run.smithi130:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:42.583 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:40:42.583 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:42.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T19:40:42.583 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T19:40:42.624 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:40:42.624 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:42.679 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T19:40:42.679 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T19:40:42.679 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:42.734 INFO:teuthology.orchestra.run.smithi130.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T19:40:42.736 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:42.796 INFO:teuthology.orchestra.run.smithi163.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T19:40:42.798 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T19:40:42.799 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T19:40:42.799 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:42.841 INFO:teuthology.orchestra.run.smithi130.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T19:40:42.856 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:42.900 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T19:40:42.915 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T19:40:42.915 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi130.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi163.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T19:40:42.915 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:42.942 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:42.942 INFO:tasks.ceph:role: osd.0 2024-03-25T19:40:42.942 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:40:42.942 DEBUG:teuthology.orchestra.run.smithi130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:43.016 INFO:teuthology.orchestra.run.smithi130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:43.487 INFO:teuthology.orchestra.run.smithi130.stdout:Discarding blocks...Done. 2024-03-25T19:40:43.489 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi130.front.sepia.ceph.com -o noatime 2024-03-25T19:40:43.490 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:43.624 DEBUG:teuthology.orchestra.run.smithi130:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:43.657 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:43.725 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:43.725 INFO:tasks.ceph:role: osd.1 2024-03-25T19:40:43.725 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:40:43.725 DEBUG:teuthology.orchestra.run.smithi130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:43.799 INFO:teuthology.orchestra.run.smithi130.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:43.800 INFO:teuthology.orchestra.run.smithi130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:44.259 INFO:teuthology.orchestra.run.smithi130.stdout:Discarding blocks...Done. 2024-03-25T19:40:44.261 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi130.front.sepia.ceph.com -o noatime 2024-03-25T19:40:44.261 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:44.345 DEBUG:teuthology.orchestra.run.smithi130:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:44.417 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:44.485 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:44.485 INFO:tasks.ceph:role: osd.2 2024-03-25T19:40:44.485 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:40:44.485 DEBUG:teuthology.orchestra.run.smithi130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:44.560 INFO:teuthology.orchestra.run.smithi130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:45.071 INFO:teuthology.orchestra.run.smithi130.stdout:Discarding blocks...Done. 2024-03-25T19:40:45.072 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi130.front.sepia.ceph.com -o noatime 2024-03-25T19:40:45.073 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:45.208 DEBUG:teuthology.orchestra.run.smithi130:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:45.240 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:45.308 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:45.309 INFO:tasks.ceph:role: osd.3 2024-03-25T19:40:45.309 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:40:45.309 DEBUG:teuthology.orchestra.run.smithi130:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:40:45.383 INFO:teuthology.orchestra.run.smithi130.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:45.383 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:45.384 INFO:teuthology.orchestra.run.smithi130.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:45.898 INFO:teuthology.orchestra.run.smithi130.stdout:Discarding blocks...Done. 2024-03-25T19:40:45.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi130.front.sepia.ceph.com -o noatime 2024-03-25T19:40:45.900 DEBUG:teuthology.orchestra.run.smithi130:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:45.999 DEBUG:teuthology.orchestra.run.smithi130:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:46.036 DEBUG:teuthology.orchestra.run.smithi130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.131+0000 7fb89f514640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.131+0000 7fb89f514640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.131+0000 7fb89f514640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.131+0000 7fb89f514640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.131+0000 7fb89f514640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T19:40:46.132 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:46.132+0000 7fb89f514640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T19:40:48.974 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T19:40:49.005 DEBUG:teuthology.orchestra.run.smithi130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.097+0000 7fa3c8e1f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.097+0000 7fa3c8e1f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.097+0000 7fa3c8e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.097+0000 7fa3c8e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:49.097 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.097+0000 7fa3c8e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T19:40:49.098 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:49.098+0000 7fa3c8e1f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T19:40:51.934 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T19:40:51.966 DEBUG:teuthology.orchestra.run.smithi130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.057+0000 7f1ceb215640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.057+0000 7f1ceb215640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.057+0000 7f1ceb215640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.057+0000 7f1ceb215640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.057+0000 7f1ceb215640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T19:40:52.057 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:52.058+0000 7f1ceb215640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T19:40:54.902 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T19:40:54.933 DEBUG:teuthology.orchestra.run.smithi130:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.024+0000 7f4d4cebf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.025+0000 7f4d4cebf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.025+0000 7f4d4cebf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.025+0000 7f4d4cebf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.025+0000 7f4d4cebf640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T19:40:55.026 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:40:55.025+0000 7f4d4cebf640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T19:40:57.866 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T19:40:57.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:57.928 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:57.928 INFO:tasks.ceph:role: osd.4 2024-03-25T19:40:57.929 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:40:57.929 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:58.006 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:58.007 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:58.524 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-03-25T19:40:58.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-03-25T19:40:58.526 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:58.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T19:40:58.720 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:58.790 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:58.790 INFO:tasks.ceph:role: osd.5 2024-03-25T19:40:58.790 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:40:58.790 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T19:40:58.866 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:58.866 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:58.866 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:58.866 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:58.866 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:58.867 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:58.867 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:58.867 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:58.867 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:58.867 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:40:59.323 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-03-25T19:40:59.325 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-03-25T19:40:59.325 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:59.424 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T19:40:59.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T19:40:59.526 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:40:59.526 INFO:tasks.ceph:role: osd.6 2024-03-25T19:40:59.527 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:40:59.527 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:40:59.602 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:40:59.603 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:40:59.603 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:00.082 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-03-25T19:41:00.084 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-03-25T19:41:00.084 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:00.242 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:00.277 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:00.345 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T19:41:00.346 INFO:tasks.ceph:role: osd.7 2024-03-25T19:41:00.346 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:41:00.346 DEBUG:teuthology.orchestra.run.smithi163:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T19:41:00.419 INFO:teuthology.orchestra.run.smithi163.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout: = sunit=0 swidth=0 blks 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T19:41:00.420 INFO:teuthology.orchestra.run.smithi163.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T19:41:00.924 INFO:teuthology.orchestra.run.smithi163.stdout:Discarding blocks...Done. 2024-03-25T19:41:00.926 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi163.front.sepia.ceph.com -o noatime 2024-03-25T19:41:00.926 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:01.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:01.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:01.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.199+0000 7ff4f3d47640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T19:41:01.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.200+0000 7ff4f3d47640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T19:41:01.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.200+0000 7ff4f3d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:01.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.200+0000 7ff4f3d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:01.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.200+0000 7ff4f3d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T19:41:01.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:01.201+0000 7ff4f3d47640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T19:41:04.049 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T19:41:04.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.164+0000 7f165dc22640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T19:41:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.165+0000 7f165dc22640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T19:41:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.165+0000 7f165dc22640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:04.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.165+0000 7f165dc22640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.165+0000 7f165dc22640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T19:41:04.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:04.165+0000 7f165dc22640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T19:41:07.006 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T19:41:07.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:07.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.125+0000 7ff3f909b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T19:41:07.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.125+0000 7ff3f909b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T19:41:07.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.125+0000 7ff3f909b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:07.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.125+0000 7ff3f909b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:07.125 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.125+0000 7ff3f909b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T19:41:07.126 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:07.126+0000 7ff3f909b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T19:41:09.976 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T19:41:10.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:10.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.098+0000 7f76e8f23640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T19:41:10.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.099+0000 7f76e8f23640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T19:41:10.098 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.099+0000 7f76e8f23640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:10.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.099+0000 7f76e8f23640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:10.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.099+0000 7f76e8f23640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T19:41:10.099 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:10.099+0000 7f76e8f23640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T19:41:12.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T19:41:12.969 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T19:41:12.970 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:12.970 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T19:41:13.001 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:13.001 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T19:41:13.072 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:13.072 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T19:41:13.141 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:13.142 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T19:41:13.210 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:13.210 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T19:41:13.278 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T19:41:13.309 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.309 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T19:41:13.382 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.382 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T19:41:13.452 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.452 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T19:41:13.521 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.521 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T19:41:13.589 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:13.589 DEBUG:teuthology.orchestra.run.smithi130:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T19:41:13.606 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:13.606 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T19:41:13.645 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T19:41:13.645 DEBUG:teuthology.orchestra.run.smithi130:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:41:13.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T19:41:13.688 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr.y] 2024-03-25T19:41:13.688 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQA60wFmWSPWKxAAOYLOU7GthmcZpJfGTpgVPQ== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.0] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQA+0wFmz23wBxAACV8ucIoHzjSkAJ7Gd94UjA== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.1] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBB0wFm4C3dBRAAX1ilC8R/qvNcaRZqAHi1Sg== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.2] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBE0wFmcA54AxAAg+P+JlilzrHGLVP3GQLdlg== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.3] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBH0wFmUUeGARAAn7KpyzbMSSPJ1W/eStR3og== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[mgr.x] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQA60wFm/GSOLxAASFiYMQVJBQq7aIWqNx5pGQ== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.4] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBN0wFmBO/yCxAAevJcLoSeeRKlBwbZ6gslpg== 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.5] 2024-03-25T19:41:13.689 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBQ0wFmXTbgCRAAq3rAzNEF2Wwm+AtmGLyL/A== 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.6] 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBT0wFm2v2JBxAASHnb84472EEp4dqkmRRSAA== 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout:[osd.7] 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQBW0wFm4cPzBRAAlunUHT+uVdML6YvAcU+ICw== 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout:[client.0] 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQA60wFmTdk+MhAA5kB9jbXVav390AgzB+YFLA== 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout:[client.1] 2024-03-25T19:41:13.690 INFO:teuthology.orchestra.run.smithi130.stdout: key = AQA60wFmVJe/NRAAgUsRBpHB35M60uO84NtbwQ== 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.y] 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA60wFmWSPWKxAAOYLOU7GthmcZpJfGTpgVPQ== 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.0] 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA+0wFmz23wBxAACV8ucIoHzjSkAJ7Gd94UjA== 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.1] 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBB0wFm4C3dBRAAX1ilC8R/qvNcaRZqAHi1Sg== 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.2] 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBE0wFmcA54AxAAg+P+JlilzrHGLVP3GQLdlg== 2024-03-25T19:41:13.711 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.3] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBH0wFmUUeGARAAn7KpyzbMSSPJ1W/eStR3og== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr.x] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA60wFm/GSOLxAASFiYMQVJBQq7aIWqNx5pGQ== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.4] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBN0wFmBO/yCxAAevJcLoSeeRKlBwbZ6gslpg== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.5] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBQ0wFmXTbgCRAAq3rAzNEF2Wwm+AtmGLyL/A== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.6] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBT0wFm2v2JBxAASHnb84472EEp4dqkmRRSAA== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[osd.7] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQBW0wFm4cPzBRAAlunUHT+uVdML6YvAcU+ICw== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[client.0] 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA60wFmTdk+MhAA5kB9jbXVav390AgzB+YFLA== 2024-03-25T19:41:13.712 INFO:teuthology.orchestra.run.smithi163.stdout:[client.1] 2024-03-25T19:41:13.713 INFO:teuthology.orchestra.run.smithi163.stdout: key = AQA60wFmVJe/NRAAgUsRBpHB35M60uO84NtbwQ== 2024-03-25T19:41:13.713 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:13.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:13.805 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.807 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.859 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.861 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.913 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.915 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.967 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:13.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.021 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:14.024 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T19:41:14.076 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.129 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.183 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.236 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T19:41:14.290 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:14.292 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:14.347 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:14.349 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T19:41:14.403 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T19:41:14.403 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T19:41:14.429 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:14.526 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T19:41:14.552 DEBUG:teuthology.orchestra.run.smithi130:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T19:41:14.619 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:14.715 DEBUG:teuthology.orchestra.run.smithi130:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T19:41:14.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T19:41:14.769 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T19:41:14.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T19:41:14.894 DEBUG:teuthology.orchestra.run.smithi130:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:14.896 DEBUG:teuthology.orchestra.run.smithi163:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T19:41:14.949 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T19:41:14.949 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:14.949 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T19:41:14.949 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T19:41:14.951 INFO:tasks.ceph.mon.a:Started 2024-03-25T19:41:14.951 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:14.952 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T19:41:14.952 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T19:41:14.953 INFO:tasks.ceph.mon.c:Started 2024-03-25T19:41:14.954 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T19:41:14.954 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T19:41:14.954 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T19:41:14.991 INFO:tasks.ceph.mon.b:Started 2024-03-25T19:41:14.992 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T19:41:14.992 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T19:41:14.992 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T19:41:14.995 INFO:tasks.ceph.mgr.y:Started 2024-03-25T19:41:14.995 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T19:41:14.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T19:41:14.997 INFO:tasks.ceph.mgr.x:Started 2024-03-25T19:41:14.998 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T19:41:14.998 DEBUG:teuthology.orchestra.run.smithi130:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T19:41:15.100 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.101+0000 7f0e001d6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:15.100 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.101+0000 7f0e001d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.101+0000 7f0e001d6640 10 monclient: monmap: 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 0 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:41:15.102144+0000 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:41:15.102144+0000 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:15.101 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:15.102 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.101+0000 7f0e001d6640 1 Processor -- start 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 1 -- start start 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient: init 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80a9920 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0df80b45a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0df8146450 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df806b100 con 0x7f0df80a9920 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b2940 con 0x7f0df80b45a0 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df813fec0 con 0x7f0df8146450 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0e001d6640 10 monclient(hunting): authenticate will time out at 4951.248414s 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0df813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:15.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0df814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:15.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.102+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0df8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:15.302 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.303+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0df813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:15.302 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.303+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0df814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:15.303 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.303+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0df8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:15.703 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.704+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0df814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:15.703 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.704+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0df813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:15.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:15.704+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0df8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:16.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:16.505+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0df813f7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:16.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:16.505+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0df814dc70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:16.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:16.505+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0df8145d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0df814dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0df8145d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0df813f7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:18.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0df8146450 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0df80b45a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80a9920 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df806b100 con 0x7f0df80a9920 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b2940 con 0x7f0df80b45a0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df813fec0 con 0x7f0df8146450 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:18.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.102+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:18.302 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.303+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:18.302 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.303+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:18.303 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.303+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:18.703 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.704+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:18.703 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.704+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:18.703 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:18.704+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:19.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:19.505+0000 7f0dfe773640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00c670 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:19.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:19.505+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:19.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:19.505+0000 7f0dfd771640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00c670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:21.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80a9920 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0df8146450 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0df80b45a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc010e20 con 0x7f0df80a9920 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df806b100 con 0x7f0df80b45a0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b2940 con 0x7f0df8146450 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0dfd771640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:21.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.103+0000 7f0dfe773640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:21.303 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.304+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:21.303 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.304+0000 7f0dfd771640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:21.303 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.304+0000 7f0dfe773640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:21.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.705+0000 7f0dfd771640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:21.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.705+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:21.704 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:21.705+0000 7f0dfe773640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:22.010 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:00:06.911 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:22.293 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:00:07.226 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:22.294 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:00:07.227 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:22.317 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:00:07.218 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:22.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:22.506+0000 7f0dfdf72640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00fdf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6789/0 2024-03-25T19:41:22.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:22.506+0000 7f0dfd771640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.130:6790/0 2024-03-25T19:41:22.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:22.506+0000 7f0dfe773640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.163:6789/0 2024-03-25T19:41:22.613 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:00:07.546 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:22.614 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:00:07.547 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc00fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T19:41:24.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0df80a9920 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0df80b45a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0df8146450 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df813fec0 con 0x7f0df80a9920 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc010e20 con 0x7f0df80b45a0 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df806b100 con 0x7f0df8146450 2024-03-25T19:41:24.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.103+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:24.124 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.125+0000 7f0dfd771640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 0x7f0ddc00f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:54694/0 (socket says 172.21.15.130:54694) 2024-03-25T19:41:24.124 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:24.125+0000 7f0dfd771640 1 -- 172.21.15.130:0/4228883933 learned_addr learned my addr 172.21.15.130:0/4228883933 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0df8146450 legacy=0x7f0ddc0056d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0df80b45a0 legacy=0x7f0ddc00f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0df80a9920 legacy=0x7f0ddc0105e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:27.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc00fdf0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc012080 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc015b90 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df80b2940 con 0x7f0ddc00fdf0 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc010720 con 0x7f0ddc015b90 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc010950 con 0x7f0ddc012080 2024-03-25T19:41:27.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:27.104+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:30.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:30.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:30.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:30.103 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc012080 legacy=0x7f0ddc014470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc015b90 legacy=0x7f0ddc005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc00fdf0 legacy=0x7f0ddc008e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc00f810 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0056c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc012080 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc00f680 con 0x7f0ddc0056c0 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc005b40 con 0x7f0ddc00f810 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc008c30 con 0x7f0ddc012080 2024-03-25T19:41:30.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:30.103+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc012080 legacy=0x7f0ddc005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc00f810 legacy=0x7f0ddc006510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc0056c0 legacy=0x7f0ddc00d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc008fa0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc012080 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc00f810 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc008e10 con 0x7f0ddc008fa0 2024-03-25T19:41:33.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc00f420 con 0x7f0ddc00f810 2024-03-25T19:41:33.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0154d0 con 0x7f0ddc012080 2024-03-25T19:41:33.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:33.103+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc012080 legacy=0x7f0ddc005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc00f810 legacy=0x7f0ddc006510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc008fa0 legacy=0x7f0ddc00ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:36.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc014470 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc014fc0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc018fb0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc015700 con 0x7f0ddc014fc0 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc00ed10 con 0x7f0ddc018fb0 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc00ef40 con 0x7f0ddc014470 2024-03-25T19:41:36.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:36.103+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:39.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:39.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:39.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc014470 legacy=0x7f0ddc0148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc018fb0 legacy=0x7f0ddc0046b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc014fc0 legacy=0x7f0ddc00e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc004dc0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc0051a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc014470 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc00f170 con 0x7f0ddc004dc0 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc014850 con 0x7f0ddc0051a0 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc0149e0 con 0x7f0ddc014470 2024-03-25T19:41:39.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:39.104+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:42.104 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): tick 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 monclient(hunting): continuing hunt 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc014470 legacy=0x7f0ddc004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc0051a0 legacy=0x7f0ddc008500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc004dc0 legacy=0x7f0ddc007df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc016fa0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc0051a0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc014470 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc014b70 con 0x7f0ddc016fa0 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc014da0 con 0x7f0ddc0051a0 2024-03-25T19:41:42.105 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ddc018aa0 con 0x7f0ddc014470 2024-03-25T19:41:42.106 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:42.105+0000 7f0def7fe640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.205 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1467307793 0 0) 0x7f0ddc014b70 con 0x7f0ddc016fa0 2024-03-25T19:41:43.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 10 monclient(hunting): my global_id is 4101 2024-03-25T19:41:43.206 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.206+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0de4003520 con 0x7f0ddc016fa0 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794970640 0 0) 0x7f0ddc014da0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 10 monclient(hunting): my global_id is 4105 2024-03-25T19:41:43.239 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.239+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc014b70 con 0x7f0ddc0051a0 2024-03-25T19:41:43.257 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37800224 0 0) 0x7f0ddc018aa0 con 0x7f0ddc014470 2024-03-25T19:41:43.257 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.258 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.258 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.258 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 10 monclient(hunting): my global_id is 4106 2024-03-25T19:41:43.258 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.258+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc014da0 con 0x7f0ddc014470 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 307377025 0 0) 0x7f0de4003520 con 0x7f0ddc016fa0 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc014470 legacy=0x7f0ddc004690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc0051a0 legacy=0x7f0ddc008500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 monclient: found mon.noname-a 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ddc018cd0 con 0x7f0ddc016fa0 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.275 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0df4003ff0 con 0x7f0ddc016fa0 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0dfcf70640 10 monclient: dump: 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0e001d6640 5 monclient: authenticate success, global_id 4101 2024-03-25T19:41:43.276 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.275+0000 7f0e001d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:43.309 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df40041c0 con 0x7f0ddc016fa0 2024-03-25T19:41:43.309 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.309 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 1 -- 172.21.15.130:0/4228883933 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0df40045d0 con 0x7f0ddc016fa0 2024-03-25T19:41:43.309 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0dfcf70640 10 monclient: dump: 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0e001d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:43.310 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0e001d6640 4 set_mon_vals no callback set 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.310+0000 7f0e001d6640 10 monclient: shutdown 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 1 -- 172.21.15.130:0/4228883933 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc016fa0 legacy=0x7f0ddc017380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 1 -- 172.21.15.130:0/4228883933 shutdown_connections 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 1 -- 172.21.15.130:0/4228883933 >> 172.21.15.130:0/4228883933 conn(0x7f0df80b7f40 msgr2=0x7f0df80b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 1 -- 172.21.15.130:0/4228883933 shutdown_connections 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 1 -- 172.21.15.130:0/4228883933 wait complete. 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 10 monclient: build_initial_monmap 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.311+0000 7f0e001d6640 10 monclient: monmap: 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 0 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:43.313 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 0.000000 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:created 0.000000 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 1 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 librados: starting msgr at 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 librados: starting objecter 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 Processor -- start 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 -- start start 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 librados: setting wanted keys 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 1 librados: calling monclient init 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 10 monclient: init 2024-03-25T19:41:43.314 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.312+0000 7f0e001d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-a con 0x7f0ddc0051a0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-c con 0x7f0ddc014470 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 10 monclient(hunting): picked mon.noname-b con 0x7f0ddc016fa0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df8141cc0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df8143140 con 0x7f0ddc016fa0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0df81432d0 con 0x7f0ddc014470 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0e001d6640 10 monclient(hunting): authenticate will time out at 4979.458680s 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0dfe773640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0ddc016fa0 0x7f0df81e1d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:40052/0 (socket says 172.21.15.130:40052) 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0dfe773640 1 -- 172.21.15.130:0/1353299426 learned_addr learned my addr 172.21.15.130:0/1353299426 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.313+0000 7f0dee7fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc016fa0 2024-03-25T19:41:43.315 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.314+0000 7f0dee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.330 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.331+0000 7f0dee7fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc014470 2024-03-25T19:41:43.330 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.331+0000 7f0dee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.363 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.363+0000 7f0dee7fc640 10 client.?.objecter ms_handle_connect 0x7f0ddc0051a0 2024-03-25T19:41:43.363 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.363+0000 7f0dee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084117011 0 0) 0x7f0ddc018cd0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 10 monclient(hunting): my global_id is 4107 2024-03-25T19:41:43.370 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.370+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc018a90 con 0x7f0ddc0051a0 2024-03-25T19:41:43.386 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778204347 0 0) 0x7f0ddc018a90 con 0x7f0ddc0051a0 2024-03-25T19:41:43.386 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ddc018cd0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.386 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0df40044d0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:43.387 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.387+0000 7f0dee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869603407 0 0) 0x7f0ddc014b70 con 0x7f0ddc014470 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 10 monclient(hunting): my global_id is 4109 2024-03-25T19:41:43.397 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.398+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ddc018a90 con 0x7f0ddc014470 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1045043214 0 0) 0x7f0ddc018cd0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 >> v1:172.21.15.130:6790/0 conn(0x7f0ddc014470 legacy=0x7f0df81daa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 >> v1:172.21.15.163:6789/0 conn(0x7f0ddc016fa0 legacy=0x7f0df81e1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 1 monclient: found mon.a 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.406 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df8143460 con 0x7f0ddc0051a0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0dee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.405+0000 7f0e001d6640 5 monclient: authenticate success, global_id 4107 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 10 monclient: _renew_subs 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df8141cc0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 10 monclient: _renew_subs 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df8141cc0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.406+0000 7f0e001d6640 1 librados: init done 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.407+0000 7f0e001d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.407 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.407+0000 7f0e001d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.408 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.407+0000 7f0e001d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.408 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.407+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df8141cc0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.618 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0df40048e0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0df4004cb0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 10 monclient: dump: 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:43.619 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0deffff640 4 set_mon_vals no callback set 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0df4005260 con 0x7f0ddc0051a0 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 2670015028 0 0) 0x7f0df4006770 con 0x7f0ddc0051a0 2024-03-25T19:41:43.620 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 10 client.4107.objecter ms_dispatch 0x7f0df8000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.618+0000 7f0dee7fc640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.619+0000 7f0dee7fc640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.619+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0df40069d0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.619+0000 7f0dee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:43.621 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.619+0000 7f0dee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:43.692 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.693+0000 7f0e001d6640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:41:43.692 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.693+0000 7f0e001d6640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:41:43.692 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.693+0000 7f0e001d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:43.693 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.693+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0df804f730 con 0x7f0ddc0051a0 2024-03-25T19:41:43.886 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:43.887+0000 7fcf0ce4b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:41:43.893 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.893+0000 7f0dee7fc640 1 -- 172.21.15.130:0/1353299426 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f0df40055f0 con 0x7f0ddc0051a0 2024-03-25T19:41:43.893 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.893+0000 7f0dee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T19:41:43.893 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.893+0000 7f0dee7fc640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T19:41:43.893 INFO:teuthology.orchestra.run.smithi130.stderr:adjusted tunables profile to default 2024-03-25T19:41:43.894 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 10 librados: watch_flush enter 2024-03-25T19:41:43.894 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 10 librados: watch_flush exit 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 10 client.4107.objecter successfully canceled tick 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 10 monclient: shutdown 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 >> v1:172.21.15.130:6789/0 conn(0x7f0ddc0051a0 legacy=0x7f0df8141380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 shutdown_connections 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 >> 172.21.15.130:0/1353299426 conn(0x7f0df80b7f40 msgr2=0x7f0df80b7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 shutdown_connections 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.894+0000 7f0e001d6640 1 -- 172.21.15.130:0/1353299426 wait complete. 2024-03-25T19:41:43.895 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:43.895+0000 7f0e001d6640 1 librados: shutdown 2024-03-25T19:41:43.905 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T19:41:43.905 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T19:41:43.905 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:43.905 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T19:41:43.932 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:43.933 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T19:41:43.944 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:43.944+0000 7fcf0ce4b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:41:44.000 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:44.000 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T19:41:44.069 DEBUG:teuthology.orchestra.run.smithi130:> set -ex 2024-03-25T19:41:44.069 DEBUG:teuthology.orchestra.run.smithi130:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T19:41:44.138 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:44.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T19:41:44.165 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:44.165 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T19:41:44.202 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.203+0000 7f011fc45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T19:41:44.232 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:44.232 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T19:41:44.264 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.264+0000 7f011fc45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T19:41:44.300 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-03-25T19:41:44.300 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T19:41:44.367 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 874157ee-ff52-4d50-bd9d-aa9258173515 0 2024-03-25T19:41:44.499 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.499+0000 7fcf0ce4b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:41:44.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.509+0000 7f35ce53b640 10 monclient: get_monmap_and_config 2024-03-25T19:41:44.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.509+0000 7f35ce53b640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.509+0000 7f35ce53b640 10 monclient: monmap: 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:44.510278+0000 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:44.510278+0000 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:44.509 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 1 Processor -- start 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 1 -- start start 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient: init 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-c con 0x7f35c80b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-b con 0x7f35c8142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-a con 0x7f35c80a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c80a9d60 con 0x7f35c80a96b0 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c813ff10 con 0x7f35c80b4700 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c80aa1f0 con 0x7f35c8142a20 2024-03-25T19:41:44.511 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.510+0000 7f35ce53b640 10 monclient(hunting): authenticate will time out at 4960.828212s 2024-03-25T19:41:44.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.511+0000 7f35c7fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f35c80b4700 0x7f35c8142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54762/0 (socket says 172.21.15.163:54762) 2024-03-25T19:41:44.512 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.511+0000 7f35c7fff640 1 -- 172.21.15.163:0/2335680366 learned_addr learned my addr 172.21.15.163:0/2335680366 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084019464 0 0) 0x7f35c80a9d60 con 0x7f35c80a96b0 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): my global_id is 4116 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b0003520 con 0x7f35c80a96b0 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3911906346 0 0) 0x7f35c813ff10 con 0x7f35c80b4700 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.513 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): my global_id is 4114 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c80a9d60 con 0x7f35c80b4700 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079728092 0 0) 0x7f35c80aa1f0 con 0x7f35c8142a20 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 10 monclient(hunting): my global_id is 4112 2024-03-25T19:41:44.514 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.513+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c813ff10 con 0x7f35c8142a20 2024-03-25T19:41:44.516 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3112854481 0 0) 0x7f35c80a9d60 con 0x7f35c80b4700 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 >> v1:172.21.15.130:6790/0 conn(0x7f35c8142a20 legacy=0x7f35c813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 >> v1:172.21.15.130:6789/0 conn(0x7f35c80a96b0 legacy=0x7f35c813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80b4ae0 con 0x7f35c80b4700 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f35b8001670 con 0x7f35c80b4700 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.517 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35c6ffd640 10 monclient: dump: 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35ce53b640 5 monclient: authenticate success, global_id 4114 2024-03-25T19:41:44.518 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.516+0000 7f35ce53b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b80017d0 con 0x7f35c80b4700 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 1 -- 172.21.15.163:0/2335680366 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f35b8001be0 con 0x7f35c80b4700 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35c6ffd640 10 monclient: dump: 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.543 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35ce53b640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:44.544 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.543+0000 7f35ce53b640 4 set_mon_vals no callback set 2024-03-25T19:41:44.546 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 10 monclient: shutdown 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 1 -- 172.21.15.163:0/2335680366 >> v1:172.21.15.163:6789/0 conn(0x7f35c80b4700 legacy=0x7f35c8142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 1 -- 172.21.15.163:0/2335680366 shutdown_connections 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 1 -- 172.21.15.163:0/2335680366 >> 172.21.15.163:0/2335680366 conn(0x7f35c80668e0 msgr2=0x7f35c8066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 1 -- 172.21.15.163:0/2335680366 shutdown_connections 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.544+0000 7f35ce53b640 1 -- 172.21.15.163:0/2335680366 wait complete. 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 10 monclient: monmap: 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:44.547 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 librados: starting msgr at 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 librados: starting objecter 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 Processor -- start 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 -- start start 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 librados: setting wanted keys 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 1 librados: calling monclient init 2024-03-25T19:41:44.548 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.545+0000 7f35ce53b640 10 monclient: init 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-a con 0x7f35c80a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-b con 0x7f35c80b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient(hunting): picked mon.noname-c con 0x7f35c8142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c812d9a0 con 0x7f35c80a96b0 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c8137700 con 0x7f35c80b4700 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c81379e0 con 0x7f35c8142a20 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.546+0000 7f35ce53b640 10 monclient(hunting): authenticate will time out at 4960.863942s 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c7fff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f35c80a96b0 0x7f35c8133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.163:39840/0 (socket says 172.21.15.163:39840) 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c7fff640 1 -- 172.21.15.163:0/3745194140 learned_addr learned my addr 172.21.15.163:0/3745194140 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter ms_handle_connect 0x7f35c80a96b0 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter ms_handle_connect 0x7f35c8142a20 2024-03-25T19:41:44.549 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter ms_handle_connect 0x7f35c80b4700 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.547+0000 7f35c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735132768 0 0) 0x7f35c81379e0 con 0x7f35c8142a20 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): my global_id is 4115 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35b4004eb0 con 0x7f35c8142a20 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1560359781 0 0) 0x7f35c812d9a0 con 0x7f35c80a96b0 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.551 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): my global_id is 4119 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c81379e0 con 0x7f35c80a96b0 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1493924322 0 0) 0x7f35c8137700 con 0x7f35c80b4700 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 10 monclient(hunting): my global_id is 4117 2024-03-25T19:41:44.552 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.549+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c812d9a0 con 0x7f35c80b4700 2024-03-25T19:41:44.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2539057178 0 0) 0x7f35c81379e0 con 0x7f35c80a96b0 2024-03-25T19:41:44.553 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35c8137700 con 0x7f35c80a96b0 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f35b8001ba0 con 0x7f35c80a96b0 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.554 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056092854 0 0) 0x7f35c812d9a0 con 0x7f35c80b4700 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.554+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35c81379e0 con 0x7f35c80b4700 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.555+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f35bc0018f0 con 0x7f35c80b4700 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.555+0000 7f35c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.555+0000 7f35c4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:44.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.555+0000 7f35c4ff9640 10 monclient(hunting): dump: 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.556 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.557 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.556+0000 7fcf0ce4b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 757585343 0 0) 0x7f35c8137700 con 0x7f35c80a96b0 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 >> v1:172.21.15.130:6790/0 conn(0x7f35c8142a20 legacy=0x7f35c812d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 >> v1:172.21.15.163:6789/0 conn(0x7f35c80b4700 legacy=0x7f35c812cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 1 monclient: found mon.a 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8136550 con 0x7f35c80a96b0 2024-03-25T19:41:44.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 5 monclient: authenticate success, global_id 4119 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 10 monclient: _renew_subs 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c8133780 con 0x7f35c80a96b0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 10 monclient: _renew_subs 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c8135cb0 con 0x7f35c80a96b0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.556+0000 7f35ce53b640 1 librados: init done 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.557+0000 7f35ce53b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.557+0000 7f35ce53b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.557+0000 7f35ce53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:44.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.557+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c804f730 con 0x7f35c80a96b0 2024-03-25T19:41:44.602 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.602+0000 7fcf0ce4b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:41:44.653 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f35b8004a90 con 0x7f35c80a96b0 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f35b8004da0 con 0x7f35c80a96b0 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 10 monclient: dump: 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.654 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.655 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.655 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.655 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f35b8005290 con 0x7f35c80a96b0 2024-03-25T19:41:44.655 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.654+0000 7f35c67fc640 4 set_mon_vals no callback set 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 4269897122 0 0) 0x7f35b80066e0 con 0x7f35c80a96b0 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 10 client.4119.objecter ms_dispatch 0x7f35c8000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f35b80069d0 con 0x7f35c80a96b0 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.656 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.655+0000 7f35c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:44.656 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.655+0000 7fcf0ce4b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:41:44.699 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.699+0000 7fcf0ce4b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:41:44.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.727+0000 7f35ce53b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "874157ee-ff52-4d50-bd9d-aa9258173515", "id": 0}] 2024-03-25T19:41:44.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.727+0000 7f35ce53b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "874157ee-ff52-4d50-bd9d-aa9258173515", "id": 0}] 2024-03-25T19:41:44.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.727+0000 7f35ce53b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:44.727 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.727+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd new", "uuid": "874157ee-ff52-4d50-bd9d-aa9258173515", "id": 0} v 0) v1 -- 0x7f35c80b7610 con 0x7f35c80a96b0 2024-03-25T19:41:44.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.811+0000 7f35c4ff9640 1 -- 172.21.15.163:0/3745194140 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "874157ee-ff52-4d50-bd9d-aa9258173515", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3103776206 0 732707515) 0x7f35b80057f0 con 0x7f35c80a96b0 2024-03-25T19:41:44.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.811+0000 7f35c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "874157ee-ff52-4d50-bd9d-aa9258173515", "id": 0}] 2024-03-25T19:41:44.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.811+0000 7f35c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:44.811 INFO:teuthology.orchestra.run.smithi163.stdout:0 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 10 librados: watch_flush enter 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 10 librados: watch_flush exit 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 10 client.4119.objecter successfully canceled tick 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 10 monclient: shutdown 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:44.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 >> v1:172.21.15.130:6789/0 conn(0x7f35c80a96b0 legacy=0x7f35c8133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 shutdown_connections 2024-03-25T19:41:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 >> 172.21.15.163:0/3745194140 conn(0x7f35c80668e0 msgr2=0x7f35c80b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 shutdown_connections 2024-03-25T19:41:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 -- 172.21.15.163:0/3745194140 wait complete. 2024-03-25T19:41:44.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.812+0000 7f35ce53b640 1 librados: shutdown 2024-03-25T19:41:44.813 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.812+0000 7f011fc45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T19:41:44.815 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.816+0000 7fcf0ce4b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:41:44.822 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new d9cf0e41-1e91-411e-bf6c-05195eff515a 1 2024-03-25T19:41:44.870 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.871+0000 7f011fc45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T19:41:44.890 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.890+0000 7fcf0ce4b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:44.915 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.916+0000 7f011fc45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.924+0000 7fbd23bc1640 10 monclient: get_monmap_and_config 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.925+0000 7fbd23bc1640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.925+0000 7fbd23bc1640 10 monclient: monmap: 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:44.925960+0000 2024-03-25T19:41:44.924 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:44.925960+0000 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:44.925 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.925+0000 7fbd23bc1640 1 Processor -- start 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.925+0000 7fbd23bc1640 1 -- start start 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.925+0000 7fbd23bc1640 10 monclient: init 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1c0b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1c142a20 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1c0a96b0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:44.926 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c0a9d60 con 0x7fbd1c142a20 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c13ff10 con 0x7fbd1c0b4700 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c0aa1f0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd23bc1640 10 monclient(hunting): authenticate will time out at 4961.243931s 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd2195d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbd1c0b4700 0x7fbd1c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54772/0 (socket says 172.21.15.163:54772) 2024-03-25T19:41:44.927 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.926+0000 7fbd2195d640 1 -- 172.21.15.163:0/350392058 learned_addr learned my addr 172.21.15.163:0/350392058 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:44.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202577446 0 0) 0x7fbd1c13ff10 con 0x7fbd1c0b4700 2024-03-25T19:41:44.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.928 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): my global_id is 4120 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd08003520 con 0x7fbd1c0b4700 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 761700491 0 0) 0x7fbd1c0aa1f0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): my global_id is 4118 2024-03-25T19:41:44.929 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c13ff10 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2300010895 0 0) 0x7fbd1c0a9d60 con 0x7fbd1c142a20 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 10 monclient(hunting): my global_id is 4122 2024-03-25T19:41:44.930 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.929+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c0aa1f0 con 0x7fbd1c142a20 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2738444016 0 0) 0x7fbd08003520 con 0x7fbd1c0b4700 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 >> v1:172.21.15.130:6790/0 conn(0x7fbd1c0a96b0 legacy=0x7fbd1c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 >> v1:172.21.15.130:6789/0 conn(0x7fbd1c142a20 legacy=0x7fbd1c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 monclient: found mon.noname-c 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c0b4ae0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.932 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd23bc1640 5 monclient: authenticate success, global_id 4120 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd18001690 con 0x7fbd1c0b4700 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd23bc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd2095b640 10 monclient: dump: 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.933 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.932+0000 7fbd23bc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd180017f0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 1 -- 172.21.15.163:0/350392058 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd18001c00 con 0x7fbd1c0b4700 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd2095b640 10 monclient: dump: 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.934 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd23bc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:44.935 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.934+0000 7fbd23bc1640 4 set_mon_vals no callback set 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 10 monclient: shutdown 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 1 -- 172.21.15.163:0/350392058 >> v1:172.21.15.163:6789/0 conn(0x7fbd1c0b4700 legacy=0x7fbd1c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 1 -- 172.21.15.163:0/350392058 shutdown_connections 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 1 -- 172.21.15.163:0/350392058 >> 172.21.15.163:0/350392058 conn(0x7fbd1c0668e0 msgr2=0x7fbd1c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 1 -- 172.21.15.163:0/350392058 shutdown_connections 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.935+0000 7fbd23bc1640 1 -- 172.21.15.163:0/350392058 wait complete. 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 10 monclient: build_initial_monmap 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 10 monclient: monmap: 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:44.938 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 librados: starting msgr at 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 librados: starting objecter 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 Processor -- start 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 -- start start 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 librados: setting wanted keys 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 1 librados: calling monclient init 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.936+0000 7fbd23bc1640 10 monclient: init 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-c con 0x7fbd1c0a96b0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:44.939 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-b con 0x7fbd1c0b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient(hunting): picked mon.noname-a con 0x7fbd1c142a20 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c13afa0 con 0x7fbd1c142a20 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c0ac7a0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd1c0adec0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd23bc1640 10 monclient(hunting): authenticate will time out at 4961.254907s 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd2115c640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fbd1c0b4700 0x7fbd1c0accb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54780/0 (socket says 172.21.15.163:54780) 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.937+0000 7fbd2115c640 1 -- 172.21.15.163:0/2365762827 learned_addr learned my addr 172.21.15.163:0/2365762827 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter ms_handle_connect 0x7fbd1c0b4700 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter ms_handle_connect 0x7fbd1c0a96b0 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter ms_handle_connect 0x7fbd1c142a20 2024-03-25T19:41:44.940 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.938+0000 7fbd127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:44.941 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fcf0ce4b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:41:44.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2446036431 0 0) 0x7fbd1c0ac7a0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.941 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 10 monclient(hunting): my global_id is 4123 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.939+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd00004eb0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201680079 0 0) 0x7fbd1c0adec0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 10 monclient(hunting): my global_id is 4121 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.940+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c0ac7a0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709618830 0 0) 0x7fbd1c13afa0 con 0x7fbd1c142a20 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:44.942 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 10 monclient(hunting): my global_id is 4125 2024-03-25T19:41:44.943 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.941+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd1c0adec0 con 0x7fbd1c142a20 2024-03-25T19:41:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669382531 0 0) 0x7fbd00004eb0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd1c13afa0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.945 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd0c001a10 con 0x7fbd1c0b4700 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 10 monclient(hunting): dump: 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.946 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.946+0000 7fbd127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3222920641 0 0) 0x7fbd1c0adec0 con 0x7fbd1c142a20 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd00004eb0 con 0x7fbd1c142a20 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd14001bd0 con 0x7fbd1c142a20 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.947+0000 7fbd127fc640 10 monclient(hunting): dump: 2024-03-25T19:41:44.947 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3463161803 0 0) 0x7fbd1c0ac7a0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbd1c0adec0 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd18001b00 con 0x7fbd1c0a96b0 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.948+0000 7fbd127fc640 10 monclient(hunting): dump: 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:44.948 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:44.949 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 601266726 0 0) 0x7fbd1c13afa0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 >> v1:172.21.15.130:6790/0 conn(0x7fbd1c0a96b0 legacy=0x7fbd1c0ac090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 >> v1:172.21.15.130:6789/0 conn(0x7fbd1c142a20 legacy=0x7fbd1c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 1 monclient: found mon.b 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd1c0ae1a0 con 0x7fbd1c0b4700 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:44.951 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd23bc1640 5 monclient: authenticate success, global_id 4123 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd23bc1640 10 monclient: _renew_subs 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.949+0000 7fbd23bc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd1c0ae480 con 0x7fbd1c0b4700 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 monclient: _renew_subs 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd1c0aea80 con 0x7fbd1c0b4700 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 1 librados: init done 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:44.952 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:44.950+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd1c04f730 con 0x7fbd1c0b4700 2024-03-25T19:41:44.975 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:44.976+0000 7f011fc45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T19:41:44.994 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:44.994+0000 7fcf0ce4b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:41:45.018 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.018+0000 7f011fc45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T19:41:45.041 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.042+0000 7fcf0ce4b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:41:45.134 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.134+0000 7f011fc45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbd0c001c70 con 0x7fbd1c0b4700 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fbd0c003a60 con 0x7fbd1c0b4700 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: dump: 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.143 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbd0c004010 con 0x7fbd1c0b4700 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd13fff640 4 set_mon_vals no callback set 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2354487285 0 0) 0x7fbd0c006670 con 0x7fbd1c0b4700 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 client.4123.objecter ms_dispatch 0x7fbd1c000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 3 client.4123.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 3 client.4123.objecter handle_osd_map decoding full epoch 3 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbd0c0069f0 con 0x7fbd1c0b4700 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.144 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.143+0000 7fbd127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:45.204 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.205+0000 7f011fc45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:45.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.215+0000 7fbd23bc1640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d9cf0e41-1e91-411e-bf6c-05195eff515a", "id": 1}] 2024-03-25T19:41:45.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.215+0000 7fbd23bc1640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d9cf0e41-1e91-411e-bf6c-05195eff515a", "id": 1}] 2024-03-25T19:41:45.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.215+0000 7fbd23bc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:45.214 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.215+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d9cf0e41-1e91-411e-bf6c-05195eff515a", "id": 1} v 0) v1 -- 0x7fbd1c0aa4a0 con 0x7fbd1c0b4700 2024-03-25T19:41:45.250 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.250+0000 7f011fc45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T19:41:45.306 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.306+0000 7f011fc45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T19:41:45.332 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.333+0000 7fbd127fc640 1 -- 172.21.15.163:0/2365762827 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d9cf0e41-1e91-411e-bf6c-05195eff515a", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 490986461 0 940483276) 0x7fbd0c0057c0 con 0x7fbd1c0b4700 2024-03-25T19:41:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.333+0000 7fbd127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d9cf0e41-1e91-411e-bf6c-05195eff515a", "id": 1}] 2024-03-25T19:41:45.333 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.333+0000 7fbd127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:45.333 INFO:teuthology.orchestra.run.smithi163.stdout:1 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 10 librados: watch_flush enter 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 10 librados: watch_flush exit 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 10 client.4123.objecter successfully canceled tick 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 10 monclient: shutdown 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 >> v1:172.21.15.163:6789/0 conn(0x7fbd1c0b4700 legacy=0x7fbd1c0accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.334 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 shutdown_connections 2024-03-25T19:41:45.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 >> 172.21.15.163:0/2365762827 conn(0x7fbd1c0668e0 msgr2=0x7fbd1c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 shutdown_connections 2024-03-25T19:41:45.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 -- 172.21.15.163:0/2365762827 wait complete. 2024-03-25T19:41:45.335 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.334+0000 7fbd23bc1640 1 librados: shutdown 2024-03-25T19:41:45.344 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new c216a970-462b-4157-8e38-8f989b551f16 2 2024-03-25T19:41:45.356 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.357+0000 7f011fc45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T19:41:45.437 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.438+0000 7fcf0ce4b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.445+0000 7f1afbd96640 10 monclient: get_monmap_and_config 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.445+0000 7f1afbd96640 10 monclient: build_initial_monmap 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.445+0000 7f1afbd96640 10 monclient: monmap: 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:45.446767+0000 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:45.446767+0000 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.445 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:45.446 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:45.446 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:45.446 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.446+0000 7f1afbd96640 1 Processor -- start 2024-03-25T19:41:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.446+0000 7f1afbd96640 1 -- start start 2024-03-25T19:41:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.446+0000 7f1afbd96640 10 monclient: init 2024-03-25T19:41:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:45.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-b con 0x7f1af4070210 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-a con 0x7f1af40b06d0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-c con 0x7f1af40705f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40b0b80 con 0x7f1af40b06d0 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af40b0430 con 0x7f1af40705f0 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af4071670 con 0x7f1af4070210 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afbd96640 10 monclient(hunting): authenticate will time out at 4961.764826s 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afa333640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1af40705f0 0x7f1af414b960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54786/0 (socket says 172.21.15.163:54786) 2024-03-25T19:41:45.448 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.447+0000 7f1afa333640 1 -- 172.21.15.163:0/784810335 learned_addr learned my addr 172.21.15.163:0/784810335 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.450+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1196382137 0 0) 0x7f1af40b0b80 con 0x7f1af40b06d0 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.450+0000 7f1af8b30640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.450+0000 7f1af8b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.450+0000 7f1af8b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.450+0000 7f1af8b30640 10 monclient(hunting): my global_id is 4128 2024-03-25T19:41:45.450 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.451+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ae0003520 con 0x7f1af40b06d0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226198578 0 0) 0x7f1ae0003520 con 0x7f1af40b06d0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 >> v1:172.21.15.130:6790/0 conn(0x7f1af4070210 legacy=0x7f1af4140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 >> v1:172.21.15.163:6789/0 conn(0x7f1af40705f0 legacy=0x7f1af414b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: _finish_hunting 0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 monclient: found mon.noname-a 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af4070a60 con 0x7f1af40b06d0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: _finish_auth 0 2024-03-25T19:41:45.455 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: _check_auth_tickets 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ae4001690 con 0x7f1af40b06d0 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1af8b30640 10 monclient: dump: 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.456 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1afbd96640 5 monclient: authenticate success, global_id 4128 2024-03-25T19:41:45.457 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.455+0000 7f1afbd96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ae40017f0 con 0x7f1af40b06d0 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 1 -- 172.21.15.163:0/784810335 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ae4001b80 con 0x7f1af40b06d0 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1af8b30640 10 monclient: dump: 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.493 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1afbd96640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:45.494 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.493+0000 7f1afbd96640 4 set_mon_vals no callback set 2024-03-25T19:41:45.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 10 monclient: shutdown 2024-03-25T19:41:45.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.496 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 1 -- 172.21.15.163:0/784810335 >> v1:172.21.15.130:6789/0 conn(0x7f1af40b06d0 legacy=0x7f1af40afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 1 -- 172.21.15.163:0/784810335 shutdown_connections 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 1 -- 172.21.15.163:0/784810335 >> 172.21.15.163:0/784810335 conn(0x7f1af40b5cd0 msgr2=0x7f1af40b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 1 -- 172.21.15.163:0/784810335 shutdown_connections 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.494+0000 7f1afbd96640 1 -- 172.21.15.163:0/784810335 wait complete. 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient: build_initial_monmap 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient: monmap: 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:45.497 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 librados: starting msgr at 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 librados: starting objecter 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 Processor -- start 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 -- start start 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 librados: setting wanted keys 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 1 librados: calling monclient init 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient: init 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-c con 0x7f1af4070210 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-a con 0x7f1af40705f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.495+0000 7f1afbd96640 10 monclient(hunting): picked mon.noname-b con 0x7f1af40b06d0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afbd96640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af406d470 con 0x7f1af40705f0 2024-03-25T19:41:45.498 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afbd96640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af4141e70 con 0x7f1af40b06d0 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afbd96640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1af406c250 con 0x7f1af4070210 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afbd96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afbd96640 10 monclient(hunting): authenticate will time out at 4961.813562s 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afa333640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1af40b06d0 0x7f1af41e2e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54800/0 (socket says 172.21.15.163:54800) 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1afa333640 1 -- 172.21.15.163:0/2757501377 learned_addr learned my addr 172.21.15.163:0/2757501377 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1aea7fc640 10 client.?.objecter ms_handle_connect 0x7f1af40b06d0 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.496+0000 7f1aea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.497+0000 7f1aea7fc640 10 client.?.objecter ms_handle_connect 0x7f1af4070210 2024-03-25T19:41:45.499 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.497+0000 7f1aea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828237602 0 0) 0x7f1af4141e70 con 0x7f1af40b06d0 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): my global_id is 4126 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8004eb0 con 0x7f1af40b06d0 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826964606 0 0) 0x7f1af406c250 con 0x7f1af4070210 2024-03-25T19:41:45.500 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 monclient(hunting): my global_id is 4124 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af4141e70 con 0x7f1af4070210 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 client.?.objecter ms_handle_connect 0x7f1af40705f0 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.499+0000 7f1aea7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410900301 0 0) 0x7f1af406d470 con 0x7f1af40705f0 2024-03-25T19:41:45.501 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:45.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:45.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:45.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 10 monclient(hunting): my global_id is 4131 2024-03-25T19:41:45.502 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.501+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1af406c250 con 0x7f1af40705f0 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 331910723 0 0) 0x7f1af4141e70 con 0x7f1af4070210 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af406d470 con 0x7f1af4070210 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1af0001930 con 0x7f1af4070210 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.504 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4140850074 0 0) 0x7f1ad8004eb0 con 0x7f1af40b06d0 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1af4141e70 con 0x7f1af40b06d0 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1aec001a10 con 0x7f1af40b06d0 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:45.505 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.504+0000 7f1aea7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.506 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 559165666 0 0) 0x7f1af406d470 con 0x7f1af4070210 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 >> v1:172.21.15.163:6789/0 conn(0x7f1af40b06d0 legacy=0x7f1af41e2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 >> v1:172.21.15.130:6789/0 conn(0x7f1af40705f0 legacy=0x7f1af41daac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 1 monclient: found mon.c 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af406c3e0 con 0x7f1af4070210 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 5 monclient: authenticate success, global_id 4124 2024-03-25T19:41:45.508 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _renew_subs 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1af41305c0 con 0x7f1af4070210 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 client.4124.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _renew_subs 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1af4136620 con 0x7f1af4070210 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 1 librados: init done 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:41:45.509 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.507+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1af404f730 con 0x7f1af4070210 2024-03-25T19:41:45.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1af0003610 con 0x7f1af4070210 2024-03-25T19:41:45.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1af0003960 con 0x7f1af4070210 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 10 monclient: dump: 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:45.557 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aebfff640 4 set_mon_vals no callback set 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1af0003f10 con 0x7f1af4070210 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1883464133 0 0) 0x7f1af0005500 con 0x7f1af4070210 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 10 client.4124.objecter ms_dispatch 0x7f1af4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 3 client.4124.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 3 client.4124.objecter handle_osd_map decoding full epoch 4 2024-03-25T19:41:45.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.556+0000 7f1aea7fc640 20 client.4124.objecter dump_active .. 0 homeless 2024-03-25T19:41:45.580 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.581+0000 7fcf0ce4b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:41:45.704 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.705+0000 7fcf0ce4b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:41:45.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.724+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f1af00051f0 con 0x7f1af4070210 2024-03-25T19:41:45.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.724+0000 7f1aea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:45.724 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.724+0000 7f1aea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:45.751 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.751+0000 7fcf0ce4b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:41:45.753 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.754+0000 7f011fc45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T19:41:45.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.797+0000 7f1afbd96640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c216a970-462b-4157-8e38-8f989b551f16", "id": 2}] 2024-03-25T19:41:45.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.797+0000 7f1afbd96640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c216a970-462b-4157-8e38-8f989b551f16", "id": 2}] 2024-03-25T19:41:45.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.797+0000 7f1afbd96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:41:45.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.797+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd new", "uuid": "c216a970-462b-4157-8e38-8f989b551f16", "id": 2} v 0) v1 -- 0x7f1af4069f30 con 0x7f1af4070210 2024-03-25T19:41:45.865 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.865+0000 7fcf0ce4b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:45.898 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:45.898+0000 7f011fc45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T19:41:45.907 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.908+0000 7f1aea7fc640 1 -- 172.21.15.163:0/2757501377 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c216a970-462b-4157-8e38-8f989b551f16", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3891714304 0 216599125) 0x7f1af0004640 con 0x7f1af4070210 2024-03-25T19:41:45.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.908+0000 7f1aea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c216a970-462b-4157-8e38-8f989b551f16", "id": 2}] 2024-03-25T19:41:45.908 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.908+0000 7f1aea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:45.908 INFO:teuthology.orchestra.run.smithi163.stdout:2 2024-03-25T19:41:45.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 10 librados: watch_flush enter 2024-03-25T19:41:45.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 10 librados: watch_flush exit 2024-03-25T19:41:45.909 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 20 client.4124.objecter shutdown clearing up homeless session... 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 10 client.4124.objecter successfully canceled tick 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 10 monclient: shutdown 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 >> v1:172.21.15.130:6790/0 conn(0x7f1af4070210 legacy=0x7f1af412feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 shutdown_connections 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 >> 172.21.15.163:0/2757501377 conn(0x7f1af40b5cd0 msgr2=0x7f1af413df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 shutdown_connections 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 -- 172.21.15.163:0/2757501377 wait complete. 2024-03-25T19:41:45.910 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:45.909+0000 7f1afbd96640 1 librados: shutdown 2024-03-25T19:41:45.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 543e2e52-08be-4e00-9c0d-105ff80c5d7f 3 2024-03-25T19:41:45.966 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:45.967+0000 7fcf0ce4b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:41:46.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.019+0000 7f70b74d3640 10 monclient: get_monmap_and_config 2024-03-25T19:41:46.018 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.019+0000 7f70b74d3640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.019+0000 7f70b74d3640 10 monclient: monmap: 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:46.020227+0000 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:46.020227+0000 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:46.019 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.020 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.019+0000 7f70b74d3640 1 Processor -- start 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 1 -- start start 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient: init 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-c con 0x7f70b00b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-b con 0x7f70b0142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-a con 0x7f70b00a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00a9d60 con 0x7f70b00a96b0 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b013ff10 con 0x7f70b00b4700 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00aa1f0 con 0x7f70b0142a20 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.020+0000 7f70b74d3640 10 monclient(hunting): authenticate will time out at 4962.338260s 2024-03-25T19:41:46.021 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.021+0000 7f70b526f640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f70b00b4700 0x7f70b0142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54808/0 (socket says 172.21.15.163:54808) 2024-03-25T19:41:46.022 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.021+0000 7f70b526f640 1 -- 172.21.15.163:0/1453616819 learned_addr learned my addr 172.21.15.163:0/1453616819 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142708960 0 0) 0x7f70b00a9d60 con 0x7f70b00a96b0 2024-03-25T19:41:46.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.023 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 10 monclient(hunting): my global_id is 4134 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.023+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f709c003520 con 0x7f70b00a96b0 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457586802 0 0) 0x7f70b013ff10 con 0x7f70b00b4700 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): my global_id is 4129 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b00a9d60 con 0x7f70b00b4700 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943516581 0 0) 0x7f70b00aa1f0 con 0x7f70b0142a20 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 10 monclient(hunting): my global_id is 4127 2024-03-25T19:41:46.024 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.024+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b013ff10 con 0x7f70b0142a20 2024-03-25T19:41:46.026 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.026+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2658802844 0 0) 0x7f709c003520 con 0x7f70b00a96b0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.026+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 >> v1:172.21.15.130:6790/0 conn(0x7f70b0142a20 legacy=0x7f70b013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 >> v1:172.21.15.163:6789/0 conn(0x7f70b00b4700 legacy=0x7f70b0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 1 monclient: found mon.noname-a 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b00b4ae0 con 0x7f70b00a96b0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70a8001a10 con 0x7f70b00a96b0 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70b74d3640 5 monclient: authenticate success, global_id 4134 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70b74d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:46.027 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70a7fff640 10 monclient: dump: 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.028 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.027+0000 7f70b74d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70a8001b70 con 0x7f70b00a96b0 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70b74d3640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 1 -- 172.21.15.163:0/1453616819 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70a8001f80 con 0x7f70b00a96b0 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70b74d3640 4 set_mon_vals no callback set 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.028+0000 7f70a7fff640 10 monclient: dump: 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.031 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 10 monclient: shutdown 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 1 -- 172.21.15.163:0/1453616819 >> v1:172.21.15.130:6789/0 conn(0x7f70b00a96b0 legacy=0x7f70b013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 1 -- 172.21.15.163:0/1453616819 shutdown_connections 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 1 -- 172.21.15.163:0/1453616819 >> 172.21.15.163:0/1453616819 conn(0x7f70b00668e0 msgr2=0x7f70b0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 1 -- 172.21.15.163:0/1453616819 shutdown_connections 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.029+0000 7f70b74d3640 1 -- 172.21.15.163:0/1453616819 wait complete. 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.032 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient: monmap: 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 librados: starting msgr at 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 librados: starting objecter 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 Processor -- start 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 -- start start 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 librados: setting wanted keys 2024-03-25T19:41:46.033 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 librados: calling monclient init 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient: init 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-a con 0x7f70b00a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-c con 0x7f70b00b4700 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 10 monclient(hunting): picked mon.noname-b con 0x7f70b0142a20 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.030+0000 7f70b74d3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00ac100 con 0x7f70b00a96b0 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b74d3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b00ae050 con 0x7f70b0142a20 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b74d3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70b014e6c0 con 0x7f70b00b4700 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b74d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b74d3640 10 monclient(hunting): authenticate will time out at 4962.348545s 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b526f640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f70b00a96b0 0x7f70b00ae470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.163:39898/0 (socket says 172.21.15.163:39898) 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70b526f640 1 -- 172.21.15.163:0/3439835728 learned_addr learned my addr 172.21.15.163:0/3439835728 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.034 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70a5ffb640 10 client.?.objecter ms_handle_connect 0x7f70b00a96b0 2024-03-25T19:41:46.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70a5ffb640 10 client.?.objecter ms_handle_connect 0x7f70b00b4700 2024-03-25T19:41:46.035 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.031+0000 7f70a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.032+0000 7f70a5ffb640 10 client.?.objecter ms_handle_connect 0x7f70b0142a20 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.032+0000 7f70a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843146496 0 0) 0x7f70b00ac100 con 0x7f70b00a96b0 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7094004eb0 con 0x7f70b00a96b0 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281484551 0 0) 0x7f70b014e6c0 con 0x7f70b00b4700 2024-03-25T19:41:46.036 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 4130 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.033+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b00ac100 con 0x7f70b00b4700 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2092875759 0 0) 0x7f70b00ae050 con 0x7f70b0142a20 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 10 monclient(hunting): my global_id is 4132 2024-03-25T19:41:46.037 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.034+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70b014e6c0 con 0x7f70b0142a20 2024-03-25T19:41:46.038 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2889863531 0 0) 0x7f7094004eb0 con 0x7f70b00a96b0 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b00ae050 con 0x7f70b00a96b0 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70ac0019f0 con 0x7f70b00a96b0 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.037+0000 7f70a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.039 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 736661653 0 0) 0x7f70b00ac100 con 0x7f70b00b4700 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7094004eb0 con 0x7f70b00b4700 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70a0001870 con 0x7f70b00b4700 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.038+0000 7f70a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.040 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 316043646 0 0) 0x7f70b014e6c0 con 0x7f70b0142a20 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f70b00ac100 con 0x7f70b0142a20 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70a8001e80 con 0x7f70b0142a20 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.039+0000 7f70a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.041 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.042 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1327977725 0 0) 0x7f70b00ae050 con 0x7f70b00a96b0 2024-03-25T19:41:46.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 >> v1:172.21.15.130:6790/0 conn(0x7f70b00b4700 legacy=0x7f70b013a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 >> v1:172.21.15.163:6789/0 conn(0x7f70b0142a20 legacy=0x7f70b00ac700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.043 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 1 monclient: found mon.a 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b014e850 con 0x7f70b00a96b0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 5 monclient: authenticate success, global_id 4137 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 10 monclient: _renew_subs 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70b00ace10 con 0x7f70b00a96b0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 10 monclient: _renew_subs 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70b01e3560 con 0x7f70b00a96b0 2024-03-25T19:41:46.044 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.041+0000 7f70b74d3640 1 librados: init done 2024-03-25T19:41:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.042+0000 7f70b74d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.042+0000 7f70b74d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.042+0000 7f70b74d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.042+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70b004f730 con 0x7f70b00a96b0 2024-03-25T19:41:46.045 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f70ac0047e0 con 0x7f70b00a96b0 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f70ac004af0 con 0x7f70b00a96b0 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 10 monclient: dump: 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.046 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f70ac005020 con 0x7f70b00a96b0 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a77fe640 4 set_mon_vals no callback set 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1780349904 0 0) 0x7f70ac006610 con 0x7f70b00a96b0 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 10 client.4137.objecter ms_dispatch 0x7f70b0000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T19:41:46.047 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.046+0000 7f70a5ffb640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T19:41:46.126 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:46.127+0000 7fcf0ce4b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:46.135 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.136+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f70ac006290 con 0x7f70b00a96b0 2024-03-25T19:41:46.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.136+0000 7f70a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.136 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.136+0000 7f70a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:46.171 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:46.172+0000 7fcf0ce4b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:46.208 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.209+0000 7f70b74d3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "543e2e52-08be-4e00-9c0d-105ff80c5d7f", "id": 3}] 2024-03-25T19:41:46.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.209+0000 7f70b74d3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "543e2e52-08be-4e00-9c0d-105ff80c5d7f", "id": 3}] 2024-03-25T19:41:46.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.209+0000 7f70b74d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:46.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.209+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd new", "uuid": "543e2e52-08be-4e00-9c0d-105ff80c5d7f", "id": 3} v 0) v1 -- 0x7f70b0136b00 con 0x7f70b00a96b0 2024-03-25T19:41:46.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.253+0000 7f70a5ffb640 1 -- 172.21.15.163:0/3439835728 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "543e2e52-08be-4e00-9c0d-105ff80c5d7f", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1027825967 0 525046306) 0x7f70ac006820 con 0x7f70b00a96b0 2024-03-25T19:41:46.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.253+0000 7f70a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "543e2e52-08be-4e00-9c0d-105ff80c5d7f", "id": 3}] 2024-03-25T19:41:46.253 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.253+0000 7f70a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:46.253 INFO:teuthology.orchestra.run.smithi163.stdout:3 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.254+0000 7f70b74d3640 10 librados: watch_flush enter 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.254+0000 7f70b74d3640 10 librados: watch_flush exit 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.254+0000 7f70b74d3640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.254+0000 7f70b74d3640 10 client.4137.objecter successfully canceled tick 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 10 monclient: shutdown 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 >> v1:172.21.15.130:6789/0 conn(0x7f70b00a96b0 legacy=0x7f70b00ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 shutdown_connections 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 >> 172.21.15.163:0/3439835728 conn(0x7f70b00668e0 msgr2=0x7f70b0066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 shutdown_connections 2024-03-25T19:41:46.255 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 -- 172.21.15.163:0/3439835728 wait complete. 2024-03-25T19:41:46.256 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.255+0000 7f70b74d3640 1 librados: shutdown 2024-03-25T19:41:46.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 7fcbf4be-140e-4f80-aa30-f7799e4db30b 4 2024-03-25T19:41:46.300 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.300+0000 7f011fc45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T19:41:46.332 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:46.333+0000 7fcf0ce4b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:41:46.345 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.346+0000 7f011fc45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T19:41:46.365 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 10 monclient: get_monmap_and_config 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 10 monclient: monmap: 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:46.367035+0000 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:46.367035+0000 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.366 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 1 Processor -- start 2024-03-25T19:41:46.367 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 1 -- start start 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.366+0000 7f11d7d96640 10 monclient: init 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-b con 0x7f11d00b4700 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-c con 0x7f11d0142a20 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-a con 0x7f11d00a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00a9d60 con 0x7f11d00a96b0 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d013ff10 con 0x7f11d0142a20 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00aa1f0 con 0x7f11d00b4700 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d7d96640 10 monclient(hunting): authenticate will time out at 4962.684990s 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d5331640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f11d0142a20 0x7f11d013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54834/0 (socket says 172.21.15.163:54834) 2024-03-25T19:41:46.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.367+0000 7f11d5331640 1 -- 172.21.15.163:0/2865406406 learned_addr learned my addr 172.21.15.163:0/2865406406 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.369 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129837626 0 0) 0x7f11d013ff10 con 0x7f11d0142a20 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): my global_id is 4135 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc003520 con 0x7f11d0142a20 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416952924 0 0) 0x7f11d00aa1f0 con 0x7f11d00b4700 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 10 monclient(hunting): my global_id is 4133 2024-03-25T19:41:46.370 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.370+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d013ff10 con 0x7f11d00b4700 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609436572 0 0) 0x7f11d00a9d60 con 0x7f11d00a96b0 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 10 monclient(hunting): my global_id is 4140 2024-03-25T19:41:46.371 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.371+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d00aa1f0 con 0x7f11d00a96b0 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269594714 0 0) 0x7f11bc003520 con 0x7f11d0142a20 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 >> v1:172.21.15.130:6790/0 conn(0x7f11d00b4700 legacy=0x7f11d0142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 >> v1:172.21.15.130:6789/0 conn(0x7f11d00a96b0 legacy=0x7f11d013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 monclient: found mon.noname-c 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d00b4ae0 con 0x7f11d0142a20 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11cc001690 con 0x7f11d0142a20 2024-03-25T19:41:46.373 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d4b30640 10 monclient: dump: 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d7d96640 5 monclient: authenticate success, global_id 4135 2024-03-25T19:41:46.374 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.373+0000 7f11d7d96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:46.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.374+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc0017f0 con 0x7f11d0142a20 2024-03-25T19:41:46.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.374+0000 7f11d4b30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.374+0000 7f11d7d96640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:46.377 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.374+0000 7f11d7d96640 4 set_mon_vals no callback set 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d4b30640 1 -- 172.21.15.163:0/2865406406 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11cc001c00 con 0x7f11d0142a20 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d4b30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d4b30640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d4b30640 10 monclient: dump: 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.378 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d7d96640 10 monclient: shutdown 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d7d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d7d96640 1 -- 172.21.15.163:0/2865406406 >> v1:172.21.15.163:6789/0 conn(0x7f11d0142a20 legacy=0x7f11d013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d7d96640 1 -- 172.21.15.163:0/2865406406 shutdown_connections 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.375+0000 7f11d7d96640 1 -- 172.21.15.163:0/2865406406 >> 172.21.15.163:0/2865406406 conn(0x7f11d00668e0 msgr2=0x7f11d0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 1 -- 172.21.15.163:0/2865406406 shutdown_connections 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 1 -- 172.21.15.163:0/2865406406 wait complete. 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 10 monclient: monmap: 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:46.379 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 1 librados: starting msgr at 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.376+0000 7f11d7d96640 1 librados: starting objecter 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 Processor -- start 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 -- start start 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 librados: setting wanted keys 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 librados: calling monclient init 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient: init 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-a con 0x7f11d00a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-b con 0x7f11d00b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient(hunting): picked mon.noname-c con 0x7f11d0142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.380 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00abb60 con 0x7f11d00a96b0 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d00abdb0 con 0x7f11d00b4700 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d014e850 con 0x7f11d0142a20 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d7d96640 10 monclient(hunting): authenticate will time out at 4962.695145s 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d5331640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f11d00b4700 0x7f11d013bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:54848/0 (socket says 172.21.15.163:54848) 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.377+0000 7f11d5331640 1 -- 172.21.15.163:0/1198464082 learned_addr learned my addr 172.21.15.163:0/1198464082 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.378+0000 7f11c67fc640 10 client.?.objecter ms_handle_connect 0x7f11d00b4700 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.378+0000 7f11c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.378+0000 7f11c67fc640 10 client.?.objecter ms_handle_connect 0x7f11d0142a20 2024-03-25T19:41:46.381 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.378+0000 7f11c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.379+0000 7f11c67fc640 10 client.?.objecter ms_handle_connect 0x7f11d00a96b0 2024-03-25T19:41:46.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.379+0000 7f11c67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707865988 0 0) 0x7f11d00abdb0 con 0x7f11d00b4700 2024-03-25T19:41:46.382 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): my global_id is 4138 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11b4004eb0 con 0x7f11d00b4700 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606543948 0 0) 0x7f11d014e850 con 0x7f11d0142a20 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.380+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d00abdb0 con 0x7f11d0142a20 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063067566 0 0) 0x7f11d00abb60 con 0x7f11d00a96b0 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 10 monclient(hunting): my global_id is 4143 2024-03-25T19:41:46.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.381+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d014e850 con 0x7f11d00a96b0 2024-03-25T19:41:46.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3265874046 0 0) 0x7f11d00abdb0 con 0x7f11d0142a20 2024-03-25T19:41:46.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d00abb60 con 0x7f11d0142a20 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206601153 0 0) 0x7f11b4004eb0 con 0x7f11d00b4700 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11d00abdb0 con 0x7f11d00b4700 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11cc001bc0 con 0x7f11d00b4700 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.385 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11c8001c30 con 0x7f11d0142a20 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.385+0000 7f11c67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.386 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.387 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.387 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.387 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.387 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.387 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1721454555 0 0) 0x7f11d014e850 con 0x7f11d00a96b0 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11b4004eb0 con 0x7f11d00a96b0 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11c00018f0 con 0x7f11d00a96b0 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient(hunting): dump: 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.389 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2126510358 0 0) 0x7f11d00abdb0 con 0x7f11d00b4700 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 >> v1:172.21.15.130:6790/0 conn(0x7f11d0142a20 legacy=0x7f11d01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 >> v1:172.21.15.130:6789/0 conn(0x7f11d00a96b0 legacy=0x7f11d013b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 monclient: found mon.b 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d0139be0 con 0x7f11d00b4700 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 5 monclient: authenticate success, global_id 4138 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 10 monclient: _renew_subs 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d013c9c0 con 0x7f11d00b4700 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 10 monclient: _renew_subs 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d00ac690 con 0x7f11d00b4700 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.388+0000 7f11d7d96640 1 librados: init done 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.389+0000 7f11d7d96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.389+0000 7f11d7d96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.391 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.389+0000 7f11d7d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.392 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.389+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d013b8d0 con 0x7f11d00b4700 2024-03-25T19:41:46.466 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.466+0000 7f011fc45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc0011f0 con 0x7f11d00b4700 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11cc004a90 con 0x7f11d00b4700 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: dump: 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.492 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f11cc004fc0 con 0x7f11d00b4700 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c7fff640 4 set_mon_vals no callback set 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2862544141 0 0) 0x7f11cc0066b0 con 0x7f11d00b4700 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 client.4138.objecter ms_dispatch 0x7f11d0000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f11cc006a50 con 0x7f11d00b4700 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.493 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.492+0000 7f11c67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:46.568 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.568+0000 7f011fc45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T19:41:46.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.580+0000 7f11d7d96640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7fcbf4be-140e-4f80-aa30-f7799e4db30b", "id": 4}] 2024-03-25T19:41:46.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.581+0000 7f11d7d96640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7fcbf4be-140e-4f80-aa30-f7799e4db30b", "id": 4}] 2024-03-25T19:41:46.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.581+0000 7f11d7d96640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.581+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7fcbf4be-140e-4f80-aa30-f7799e4db30b", "id": 4} v 0) v1 -- 0x7f11d00b76e0 con 0x7f11d00b4700 2024-03-25T19:41:46.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.665+0000 7f11c67fc640 1 -- 172.21.15.163:0/1198464082 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7fcbf4be-140e-4f80-aa30-f7799e4db30b", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3110778684 0 1697012583) 0x7f11cc006270 con 0x7f11d00b4700 2024-03-25T19:41:46.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.665+0000 7f11c67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7fcbf4be-140e-4f80-aa30-f7799e4db30b", "id": 4}] 2024-03-25T19:41:46.665 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.665+0000 7f11c67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:46.665 INFO:teuthology.orchestra.run.smithi163.stdout:4 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 10 librados: watch_flush enter 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 10 librados: watch_flush exit 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 10 client.4138.objecter successfully canceled tick 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 10 monclient: shutdown 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 >> v1:172.21.15.163:6789/0 conn(0x7f11d00b4700 legacy=0x7f11d013bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 shutdown_connections 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 >> 172.21.15.163:0/1198464082 conn(0x7f11d00668e0 msgr2=0x7f11d00b2580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 shutdown_connections 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 -- 172.21.15.163:0/1198464082 wait complete. 2024-03-25T19:41:46.667 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.666+0000 7f11d7d96640 1 librados: shutdown 2024-03-25T19:41:46.676 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new ef685213-8889-4279-b9f3-5fb85b5936f7 5 2024-03-25T19:41:46.732 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.733+0000 7f011fc45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T19:41:46.774 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.775+0000 7f011fc45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.777+0000 7f3d05b9d640 10 monclient: get_monmap_and_config 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.777+0000 7f3d05b9d640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.777+0000 7f3d05b9d640 10 monclient: monmap: 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:46.778656+0000 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:46.778656+0000 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.777 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:46.778 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:46.778 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.778+0000 7f3d05b9d640 1 Processor -- start 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.778+0000 7f3d05b9d640 1 -- start start 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.778+0000 7f3d05b9d640 10 monclient: init 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d000b4700 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.779 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d00142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d000a96b0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000a9d60 con 0x7f3d000b4700 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0013ff10 con 0x7f3d000a96b0 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d000aa1f0 con 0x7f3d00142a20 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3d05b9d640 10 monclient(hunting): authenticate will time out at 4963.096747s 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3cfffff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3d000a96b0 0x7f3d0013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44360/0 (socket says 172.21.15.163:44360) 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3cff7fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f3d000b4700 0x7f3d00142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.163:55996/0 (socket says 172.21.15.163:55996) 2024-03-25T19:41:46.780 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.779+0000 7f3cfffff640 1 -- 172.21.15.163:0/3110222226 learned_addr learned my addr 172.21.15.163:0/3110222226 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949401496 0 0) 0x7f3d0013ff10 con 0x7f3d000a96b0 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): my global_id is 4141 2024-03-25T19:41:46.781 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ce8003520 con 0x7f3d000a96b0 2024-03-25T19:41:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3484981933 0 0) 0x7f3d000aa1f0 con 0x7f3d00142a20 2024-03-25T19:41:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.782 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): my global_id is 4139 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d0013ff10 con 0x7f3d00142a20 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850656368 0 0) 0x7f3d000a9d60 con 0x7f3d000b4700 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 10 monclient(hunting): my global_id is 4146 2024-03-25T19:41:46.783 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.782+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d000aa1f0 con 0x7f3d000b4700 2024-03-25T19:41:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802597031 0 0) 0x7f3d0013ff10 con 0x7f3d00142a20 2024-03-25T19:41:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 >> v1:172.21.15.163:6789/0 conn(0x7f3d000a96b0 legacy=0x7f3d0013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 >> v1:172.21.15.130:6789/0 conn(0x7f3d000b4700 legacy=0x7f3d00142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.786 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 monclient: found mon.noname-b 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d000b4ae0 con 0x7f3d00142a20 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf4001670 con 0x7f3d00142a20 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3cfe7fc640 10 monclient: dump: 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.787 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3d05b9d640 5 monclient: authenticate success, global_id 4139 2024-03-25T19:41:46.788 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.786+0000 7f3d05b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cf40017d0 con 0x7f3d00142a20 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 1 -- 172.21.15.163:0/3110222226 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf4001be0 con 0x7f3d00142a20 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3cfe7fc640 10 monclient: dump: 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.790 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3d05b9d640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.788+0000 7f3d05b9d640 4 set_mon_vals no callback set 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 10 monclient: shutdown 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 1 -- 172.21.15.163:0/3110222226 >> v1:172.21.15.130:6790/0 conn(0x7f3d00142a20 legacy=0x7f3d0013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 1 -- 172.21.15.163:0/3110222226 shutdown_connections 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 1 -- 172.21.15.163:0/3110222226 >> 172.21.15.163:0/3110222226 conn(0x7f3d000668e0 msgr2=0x7f3d00066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 1 -- 172.21.15.163:0/3110222226 shutdown_connections 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.789+0000 7f3d05b9d640 1 -- 172.21.15.163:0/3110222226 wait complete. 2024-03-25T19:41:46.791 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 10 monclient: build_initial_monmap 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 10 monclient: monmap: 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 librados: starting msgr at 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 librados: starting objecter 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 Processor -- start 2024-03-25T19:41:46.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 -- start start 2024-03-25T19:41:46.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 librados: setting wanted keys 2024-03-25T19:41:46.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 1 librados: calling monclient init 2024-03-25T19:41:46.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.790+0000 7f3d05b9d640 10 monclient: init 2024-03-25T19:41:46.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:46.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:41:46.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3d000a96b0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:46.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3d000b4700 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:46.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3d00142a20 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d0013a4b0 con 0x7f3d000b4700 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d00138e40 con 0x7f3d00142a20 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d00135860 con 0x7f3d000a96b0 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3d05b9d640 10 monclient(hunting): authenticate will time out at 4963.108921s 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3cfeffd640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f3d000b4700 0x7f3d00130ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.163:56010/0 (socket says 172.21.15.163:56010) 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.791+0000 7f3cfeffd640 1 -- 172.21.15.163:0/4097543725 learned_addr learned my addr 172.21.15.163:0/4097543725 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter ms_handle_connect 0x7f3d000b4700 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter ms_handle_connect 0x7f3d000a96b0 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter ms_handle_connect 0x7f3d00142a20 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.792+0000 7f3ce7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157386914 0 0) 0x7f3d00135860 con 0x7f3d000a96b0 2024-03-25T19:41:46.796 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): my global_id is 4142 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cec004eb0 con 0x7f3d000a96b0 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77143704 0 0) 0x7f3d00138e40 con 0x7f3d00142a20 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): my global_id is 4144 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00135860 con 0x7f3d00142a20 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856652834 0 0) 0x7f3d0013a4b0 con 0x7f3d000b4700 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:46.797 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 10 monclient(hunting): my global_id is 4149 2024-03-25T19:41:46.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.794+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d00138e40 con 0x7f3d000b4700 2024-03-25T19:41:46.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 500727963 0 0) 0x7f3d00135860 con 0x7f3d00142a20 2024-03-25T19:41:46.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d0013a4b0 con 0x7f3d00142a20 2024-03-25T19:41:46.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf80016d0 con 0x7f3d00142a20 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.799 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.799+0000 7f3ce7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:46.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723245652 0 0) 0x7f3cec004eb0 con 0x7f3d000a96b0 2024-03-25T19:41:46.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d00135860 con 0x7f3d000a96b0 2024-03-25T19:41:46.800 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf0001cb0 con 0x7f3d000a96b0 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.800+0000 7f3ce7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1832252685 0 0) 0x7f3d00138e40 con 0x7f3d000b4700 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cec004eb0 con 0x7f3d000b4700 2024-03-25T19:41:46.801 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf4001630 con 0x7f3d000b4700 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.801+0000 7f3ce7fff640 10 monclient(hunting): dump: 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.802 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1305388845 0 0) 0x7f3d0013a4b0 con 0x7f3d00142a20 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 >> v1:172.21.15.130:6790/0 conn(0x7f3d000a96b0 legacy=0x7f3d00138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 >> v1:172.21.15.130:6789/0 conn(0x7f3d000b4700 legacy=0x7f3d00130ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 1 monclient: found mon.b 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d001359f0 con 0x7f3d00142a20 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3ce7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:46.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 5 monclient: authenticate success, global_id 4144 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 10 monclient: _renew_subs 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d001315b0 con 0x7f3d00142a20 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 10 monclient: _renew_subs 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001db070 con 0x7f3d00142a20 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.802+0000 7f3d05b9d640 1 librados: init done 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.803+0000 7f3d05b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.803+0000 7f3d05b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.803+0000 7f3d05b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.805 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.803+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00139540 con 0x7f3d00142a20 2024-03-25T19:41:46.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3cf8001ba0 con 0x7f3d00142a20 2024-03-25T19:41:46.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:46.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3cf8004980 con 0x7f3d00142a20 2024-03-25T19:41:46.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 10 monclient: dump: 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3cfdffb640 4 set_mon_vals no callback set 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3cf8004e70 con 0x7f3d00142a20 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 1373308019 0 0) 0x7f3cf8006580 con 0x7f3d00142a20 2024-03-25T19:41:46.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 10 client.4144.objecter ms_dispatch 0x7f3d00000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T19:41:46.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T19:41:46.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-25T19:41:46.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.806+0000 7f3ce7fff640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-25T19:41:46.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.902+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3cf8006110 con 0x7f3d00142a20 2024-03-25T19:41:46.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.902+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:46.902 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.902+0000 7f3ce7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:46.934 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:46.934+0000 7f011fc45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T19:41:46.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.975+0000 7f3d05b9d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ef685213-8889-4279-b9f3-5fb85b5936f7", "id": 5}] 2024-03-25T19:41:46.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.975+0000 7f3d05b9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ef685213-8889-4279-b9f3-5fb85b5936f7", "id": 5}] 2024-03-25T19:41:46.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.975+0000 7f3d05b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:46.974 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:46.975+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ef685213-8889-4279-b9f3-5fb85b5936f7", "id": 5} v 0) v1 -- 0x7f3d00138fd0 con 0x7f3d00142a20 2024-03-25T19:41:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.060+0000 7f3ce7fff640 1 -- 172.21.15.163:0/4097543725 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ef685213-8889-4279-b9f3-5fb85b5936f7", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3162716804 0 1988412176) 0x7f3cf8006380 con 0x7f3d00142a20 2024-03-25T19:41:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.060+0000 7f3ce7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ef685213-8889-4279-b9f3-5fb85b5936f7", "id": 5}] 2024-03-25T19:41:47.060 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.060+0000 7f3ce7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:47.060 INFO:teuthology.orchestra.run.smithi163.stdout:5 2024-03-25T19:41:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 10 librados: watch_flush enter 2024-03-25T19:41:47.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 10 librados: watch_flush exit 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 10 client.4144.objecter successfully canceled tick 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 10 monclient: shutdown 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.061+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 >> v1:172.21.15.163:6789/0 conn(0x7f3d00142a20 legacy=0x7f3d001da960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.062+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 shutdown_connections 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.062+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 >> 172.21.15.163:0/4097543725 conn(0x7f3d000668e0 msgr2=0x7f3d000b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.062+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 shutdown_connections 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.062+0000 7f3d05b9d640 1 -- 172.21.15.163:0/4097543725 wait complete. 2024-03-25T19:41:47.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.062+0000 7f3d05b9d640 1 librados: shutdown 2024-03-25T19:41:47.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 36fb90b8-9c7b-4ac1-a2b2-0004fd9541de 6 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.177+0000 7f42350ba640 10 monclient: get_monmap_and_config 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.177+0000 7f42350ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.177+0000 7f42350ba640 10 monclient: monmap: 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.177 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:47.178461+0000 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:47.178461+0000 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 1 Processor -- start 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 1 -- start start 2024-03-25T19:41:47.178 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient: init 2024-03-25T19:41:47.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:47.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-a con 0x7f42300b4700 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:47.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4230142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:47.179 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-c con 0x7f42300a96b0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300a9d60 con 0x7f42300b4700 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423013ff10 con 0x7f42300a96b0 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300aa1f0 con 0x7f4230142a20 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.178+0000 7f42350ba640 10 monclient(hunting): authenticate will time out at 4963.496214s 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.179+0000 7f422f59e640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f42300a96b0 0x7f423013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44380/0 (socket says 172.21.15.163:44380) 2024-03-25T19:41:47.180 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.179+0000 7f422f59e640 1 -- 172.21.15.163:0/2763310421 learned_addr learned my addr 172.21.15.163:0/2763310421 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:47.180 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:47.179+0000 7fcf0ce4b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999892226 0 0) 0x7f42300aa1f0 con 0x7f4230142a20 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): my global_id is 4145 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218003520 con 0x7f4230142a20 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498386660 0 0) 0x7f423013ff10 con 0x7f42300a96b0 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.181 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 10 monclient(hunting): my global_id is 4147 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.181+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42300aa1f0 con 0x7f42300a96b0 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 736107060 0 0) 0x7f42300a9d60 con 0x7f42300b4700 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 10 monclient(hunting): my global_id is 4152 2024-03-25T19:41:47.182 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.182+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423013ff10 con 0x7f42300b4700 2024-03-25T19:41:47.184 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.184+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2468354613 0 0) 0x7f4218003520 con 0x7f4230142a20 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.184+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 >> v1:172.21.15.163:6789/0 conn(0x7f42300a96b0 legacy=0x7f423013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.184+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 >> v1:172.21.15.130:6789/0 conn(0x7f42300b4700 legacy=0x7f4230142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 1 monclient: found mon.noname-b 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42300b4ae0 con 0x7f4230142a20 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4224001690 con 0x7f4230142a20 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f42350ba640 5 monclient: authenticate success, global_id 4145 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f42350ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.185 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f422dd9b640 10 monclient: dump: 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.186 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.185+0000 7f42350ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.186+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42240017f0 con 0x7f4230142a20 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.186+0000 7f422dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.186+0000 7f42350ba640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.186+0000 7f42350ba640 4 set_mon_vals no callback set 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f422dd9b640 1 -- 172.21.15.163:0/2763310421 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4224001c40 con 0x7f4230142a20 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f422dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f422dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f422dd9b640 10 monclient: dump: 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.188 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 10 monclient: shutdown 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 1 -- 172.21.15.163:0/2763310421 >> v1:172.21.15.130:6790/0 conn(0x7f4230142a20 legacy=0x7f423013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 1 -- 172.21.15.163:0/2763310421 shutdown_connections 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 1 -- 172.21.15.163:0/2763310421 >> 172.21.15.163:0/2763310421 conn(0x7f42300668e0 msgr2=0x7f4230066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.189 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 1 -- 172.21.15.163:0/2763310421 shutdown_connections 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.187+0000 7f42350ba640 1 -- 172.21.15.163:0/2763310421 wait complete. 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 10 monclient: monmap: 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 librados: starting msgr at 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 librados: starting objecter 2024-03-25T19:41:47.190 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 Processor -- start 2024-03-25T19:41:47.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 -- start start 2024-03-25T19:41:47.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 librados: setting wanted keys 2024-03-25T19:41:47.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 1 librados: calling monclient init 2024-03-25T19:41:47.191 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.188+0000 7f42350ba640 10 monclient: init 2024-03-25T19:41:47.192 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-a con 0x7f42300a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-c con 0x7f42300b4700 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient(hunting): picked mon.noname-b con 0x7f4230142a20 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4230133f50 con 0x7f42300a96b0 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300abe40 con 0x7f4230142a20 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300ac120 con 0x7f42300b4700 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f42350ba640 10 monclient(hunting): authenticate will time out at 4963.506974s 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f422f59e640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4230142a20 0x7f4230133840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44382/0 (socket says 172.21.15.163:44382) 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.189+0000 7f422f59e640 1 -- 172.21.15.163:0/968442077 learned_addr learned my addr 172.21.15.163:0/968442077 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter ms_handle_connect 0x7f42300a96b0 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter ms_handle_connect 0x7f4230142a20 2024-03-25T19:41:47.193 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter ms_handle_connect 0x7f42300b4700 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.190+0000 7f42177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2015030810 0 0) 0x7f42300ac120 con 0x7f42300b4700 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c004eb0 con 0x7f42300b4700 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802170744 0 0) 0x7f42300abe40 con 0x7f4230142a20 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): my global_id is 4150 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42300ac120 con 0x7f4230142a20 2024-03-25T19:41:47.194 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973071117 0 0) 0x7f4230133f50 con 0x7f42300a96b0 2024-03-25T19:41:47.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T19:41:47.195 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.192+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42300abe40 con 0x7f42300a96b0 2024-03-25T19:41:47.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.196+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1711127663 0 0) 0x7f42300ac120 con 0x7f4230142a20 2024-03-25T19:41:47.196 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4230133f50 con 0x7f4230142a20 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4228001d50 con 0x7f4230142a20 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): dump: 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.197 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1396062903 0 0) 0x7f421c004eb0 con 0x7f42300b4700 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42300ac120 con 0x7f42300b4700 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4224001bc0 con 0x7f42300b4700 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): dump: 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.198 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147172352 0 0) 0x7f42300abe40 con 0x7f42300a96b0 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f421c004eb0 con 0x7f42300a96b0 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4220001830 con 0x7f42300a96b0 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.197+0000 7f42177fe640 10 monclient(hunting): dump: 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.199 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.200 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1520000620 0 0) 0x7f4230133f50 con 0x7f4230142a20 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 >> v1:172.21.15.130:6790/0 conn(0x7f42300b4700 legacy=0x7f4230133130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 >> v1:172.21.15.130:6789/0 conn(0x7f42300a96b0 legacy=0x7f423013beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 1 monclient: found mon.b 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423014e3a0 con 0x7f4230142a20 2024-03-25T19:41:47.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 5 monclient: authenticate success, global_id 4150 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 10 monclient: _renew_subs 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4230139e10 con 0x7f4230142a20 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 10 monclient: _renew_subs 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42300aca00 con 0x7f4230142a20 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.200+0000 7f42350ba640 1 librados: init done 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.201+0000 7f42350ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.201+0000 7f42350ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.201+0000 7f42350ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.202 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.201+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423004f730 con 0x7f4230142a20 2024-03-25T19:41:47.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4228004b60 con 0x7f4230142a20 2024-03-25T19:41:47.302 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4228004e70 con 0x7f4230142a20 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: dump: 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.303 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4228005360 con 0x7f4230142a20 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f422d59a640 4 set_mon_vals no callback set 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2955873850 0 0) 0x7f4228006a20 con 0x7f4230142a20 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 client.4150.objecter ms_dispatch 0x7f4230000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4228018e00 con 0x7f4230142a20 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.304 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.302+0000 7f42177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:47.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.372+0000 7f42350ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "36fb90b8-9c7b-4ac1-a2b2-0004fd9541de", "id": 6}] 2024-03-25T19:41:47.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.372+0000 7f42350ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "36fb90b8-9c7b-4ac1-a2b2-0004fd9541de", "id": 6}] 2024-03-25T19:41:47.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.372+0000 7f42350ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.372 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.372+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd new", "uuid": "36fb90b8-9c7b-4ac1-a2b2-0004fd9541de", "id": 6} v 0) v1 -- 0x7f42300b7330 con 0x7f4230142a20 2024-03-25T19:41:47.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.446+0000 7f42177fe640 1 -- 172.21.15.163:0/968442077 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36fb90b8-9c7b-4ac1-a2b2-0004fd9541de", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1307960750 0 1113811849) 0x7f4228006430 con 0x7f4230142a20 2024-03-25T19:41:47.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.446+0000 7f42177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "36fb90b8-9c7b-4ac1-a2b2-0004fd9541de", "id": 6}] 2024-03-25T19:41:47.445 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.446+0000 7f42177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:47.445 INFO:teuthology.orchestra.run.smithi163.stdout:6 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.446+0000 7f42350ba640 10 librados: watch_flush enter 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 10 librados: watch_flush exit 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 10 client.4150.objecter successfully canceled tick 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 10 monclient: shutdown 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 >> v1:172.21.15.163:6789/0 conn(0x7f4230142a20 legacy=0x7f4230133840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 shutdown_connections 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 >> 172.21.15.163:0/968442077 conn(0x7f42300668e0 msgr2=0x7f42300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 shutdown_connections 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 -- 172.21.15.163:0/968442077 wait complete. 2024-03-25T19:41:47.447 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.447+0000 7f42350ba640 1 librados: shutdown 2024-03-25T19:41:47.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --cluster ceph osd new 8cce7c73-66f0-4ffc-93c3-bde0072d6f30 7 2024-03-25T19:41:47.466 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:47.467+0000 7fcf0ce4b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:41:47.523 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:47.523+0000 7fcf0ce4b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:41:47.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 10 monclient: get_monmap_and_config 2024-03-25T19:41:47.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 10 monclient: monmap: 2024-03-25T19:41:47.559 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:47.559 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:41:47.561023+0000 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:41:47.561023+0000 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:47.560 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 1 Processor -- start 2024-03-25T19:41:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 1 -- start start 2024-03-25T19:41:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.560+0000 7f7917724640 10 monclient: init 2024-03-25T19:41:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient(hunting): picked mon.noname-c con 0x7f79100b4700 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient(hunting): picked mon.noname-b con 0x7f7910142a20 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient(hunting): picked mon.noname-a con 0x7f79100a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79100a9d60 con 0x7f79100a96b0 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791013ff10 con 0x7f79100b4700 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79100aa1f0 con 0x7f7910142a20 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f7917724640 10 monclient(hunting): authenticate will time out at 4963.878981s 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f79154c0640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f79100b4700 0x7f7910142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44384/0 (socket says 172.21.15.163:44384) 2024-03-25T19:41:47.562 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.561+0000 7f79154c0640 1 -- 172.21.15.163:0/2822883780 learned_addr learned my addr 172.21.15.163:0/2822883780 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102505017 0 0) 0x7f791013ff10 con 0x7f79100b4700 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): my global_id is 4153 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78fc003520 con 0x7f79100b4700 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491424525 0 0) 0x7f79100a9d60 con 0x7f79100a96b0 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): my global_id is 4158 2024-03-25T19:41:47.564 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f791013ff10 con 0x7f79100a96b0 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194061189 0 0) 0x7f79100aa1f0 con 0x7f7910142a20 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 10 monclient(hunting): my global_id is 4151 2024-03-25T19:41:47.565 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.564+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79100a9d60 con 0x7f7910142a20 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3708241700 0 0) 0x7f78fc003520 con 0x7f79100b4700 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 >> v1:172.21.15.130:6790/0 conn(0x7f7910142a20 legacy=0x7f791013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 >> v1:172.21.15.130:6789/0 conn(0x7f79100a96b0 legacy=0x7f791013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 monclient: found mon.noname-c 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79100b4ae0 con 0x7f79100b4700 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7917724640 5 monclient: authenticate success, global_id 4153 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f790c001690 con 0x7f79100b4700 2024-03-25T19:41:47.567 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7917724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7903fff640 10 monclient: dump: 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.568 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.567+0000 7f7917724640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f790c0017f0 con 0x7f79100b4700 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 1 -- 172.21.15.163:0/2822883780 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f790c001c00 con 0x7f79100b4700 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7917724640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7917724640 4 set_mon_vals no callback set 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.568+0000 7f7903fff640 10 monclient: dump: 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.571 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 10 monclient: shutdown 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 1 -- 172.21.15.163:0/2822883780 >> v1:172.21.15.163:6789/0 conn(0x7f79100b4700 legacy=0x7f7910142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 1 -- 172.21.15.163:0/2822883780 shutdown_connections 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 1 -- 172.21.15.163:0/2822883780 >> 172.21.15.163:0/2822883780 conn(0x7f79100668e0 msgr2=0x7f7910066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 1 -- 172.21.15.163:0/2822883780 shutdown_connections 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.569+0000 7f7917724640 1 -- 172.21.15.163:0/2822883780 wait complete. 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 10 monclient: build_initial_monmap 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 10 monclient: monmap: 2024-03-25T19:41:47.572 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 0 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 0.000000 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:created 0.000000 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 1 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 librados: starting msgr at 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 librados: starting objecter 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 Processor -- start 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 -- start start 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 librados: setting wanted keys 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 1 librados: calling monclient init 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.570+0000 7f7917724640 10 monclient: init 2024-03-25T19:41:47.573 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient(hunting): picked mon.noname-a con 0x7f79100a96b0 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient(hunting): picked mon.noname-c con 0x7f79100b4700 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient(hunting): picked mon.noname-b con 0x7f7910142a20 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79100b1850 con 0x7f79100a96b0 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79100ad150 con 0x7f7910142a20 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79100ad430 con 0x7f79100b4700 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7917724640 10 monclient(hunting): authenticate will time out at 4963.888794s 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7915cc1640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f7910142a20 0x7f79101e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.163:44394/0 (socket says 172.21.15.163:44394) 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7915cc1640 1 -- 172.21.15.163:0/3376706223 learned_addr learned my addr 172.21.15.163:0/3376706223 (peer_addr_for_me v1:172.21.15.163:0/0) 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7901ffb640 10 client.?.objecter ms_handle_connect 0x7f7910142a20 2024-03-25T19:41:47.574 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.571+0000 7f7901ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.572+0000 7f7901ffb640 10 client.?.objecter ms_handle_connect 0x7f79100a96b0 2024-03-25T19:41:47.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.572+0000 7f7901ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.572+0000 7f7901ffb640 10 client.?.objecter ms_handle_connect 0x7f79100b4700 2024-03-25T19:41:47.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.572+0000 7f7901ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:47.575 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614179492 0 0) 0x7f79100ad150 con 0x7f7910142a20 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 10 monclient(hunting): my global_id is 4156 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.573+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f4004eb0 con 0x7f7910142a20 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3178012028 0 0) 0x7f79100b1850 con 0x7f79100a96b0 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): my global_id is 4161 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79100ad150 con 0x7f79100a96b0 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549722098 0 0) 0x7f79100ad430 con 0x7f79100b4700 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:47.576 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.574+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79100b1850 con 0x7f79100b4700 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2067713251 0 0) 0x7f78f4004eb0 con 0x7f7910142a20 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79100ad430 con 0x7f7910142a20 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7908001c90 con 0x7f7910142a20 2024-03-25T19:41:47.577 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.578 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.578+0000 7f7901ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3608317502 0 0) 0x7f79100ad150 con 0x7f79100a96b0 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78f4004eb0 con 0x7f79100a96b0 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f790c001b00 con 0x7f79100a96b0 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:47.579 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501527861 0 0) 0x7f79100b1850 con 0x7f79100b4700 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79100ad150 con 0x7f79100b4700 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7904001830 con 0x7f79100b4700 2024-03-25T19:41:47.580 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.579+0000 7f7901ffb640 10 monclient(hunting): dump: 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.581 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1654340696 0 0) 0x7f79100ad430 con 0x7f7910142a20 2024-03-25T19:41:47.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 >> v1:172.21.15.130:6790/0 conn(0x7f79100b4700 legacy=0x7f79100ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.582 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 >> v1:172.21.15.130:6789/0 conn(0x7f79100a96b0 legacy=0x7f7910139db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 1 monclient: found mon.b 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79100ac180 con 0x7f7910142a20 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 10 monclient: _finish_auth 0 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7901ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7917724640 5 monclient: authenticate success, global_id 4156 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7917724640 10 monclient: _renew_subs 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7917724640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.581+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791013b5c0 con 0x7f7910142a20 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 monclient: _renew_subs 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.583 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79100abaf0 con 0x7f7910142a20 2024-03-25T19:41:47.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 1 librados: init done 2024-03-25T19:41:47.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.584 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.582+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791004f730 con 0x7f7910142a20 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7908003970 con 0x7f7910142a20 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7908003d80 con 0x7f7910142a20 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 10 monclient: dump: 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:epoch 1 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:47.585 INFO:teuthology.orchestra.run.smithi163.stderr:election_strategy: 3 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f7908004330 con 0x7f7910142a20 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2699333633 0 0) 0x7f7908006c30 con 0x7f7910142a20 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 10 client.4156.objecter ms_dispatch 0x7f7910000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f79037fe640 4 set_mon_vals no callback set 2024-03-25T19:41:47.586 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.585+0000 7f7901ffb640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-25T19:41:47.646 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:47.646+0000 7fcf0ce4b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:41:47.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.682+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f79080045e0 con 0x7f7910142a20 2024-03-25T19:41:47.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.682+0000 7f7901ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:47.682 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.682+0000 7f7901ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:47.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.752+0000 7f7917724640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8cce7c73-66f0-4ffc-93c3-bde0072d6f30", "id": 7}] 2024-03-25T19:41:47.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.752+0000 7f7917724640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8cce7c73-66f0-4ffc-93c3-bde0072d6f30", "id": 7}] 2024-03-25T19:41:47.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.752+0000 7f7917724640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:47.752 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.752+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8cce7c73-66f0-4ffc-93c3-bde0072d6f30", "id": 7} v 0) v1 -- 0x7f79100b7190 con 0x7f7910142a20 2024-03-25T19:41:47.936 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:47.936+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3009388901 0 0) 0x7f7908006680 con 0x7f7910142a20 2024-03-25T19:41:48.055 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:47.939+0000 7f011fc45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T19:41:48.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.154+0000 7f7901ffb640 1 -- 172.21.15.163:0/3376706223 <== mon.1 v1:172.21.15.163:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8cce7c73-66f0-4ffc-93c3-bde0072d6f30", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 2846119396 0 1371665406) 0x7f7908006930 con 0x7f7910142a20 2024-03-25T19:41:48.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.154+0000 7f7901ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8cce7c73-66f0-4ffc-93c3-bde0072d6f30", "id": 7}] 2024-03-25T19:41:48.154 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.154+0000 7f7901ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:48.154 INFO:teuthology.orchestra.run.smithi163.stdout:7 2024-03-25T19:41:48.155 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 10 librados: watch_flush enter 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 10 librados: watch_flush exit 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 10 client.4156.objecter successfully canceled tick 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 10 monclient: shutdown 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 >> v1:172.21.15.163:6789/0 conn(0x7f7910142a20 legacy=0x7f79101e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 shutdown_connections 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 >> 172.21.15.163:0/3376706223 conn(0x7f79100668e0 msgr2=0x7f79101364a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 shutdown_connections 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 -- 172.21.15.163:0/3376706223 wait complete. 2024-03-25T19:41:48.156 INFO:teuthology.orchestra.run.smithi163.stderr:2024-03-25T19:41:48.155+0000 7f7917724640 1 librados: shutdown 2024-03-25T19:41:48.165 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.165 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T19:41:48.166 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T19:41:48.168 INFO:tasks.ceph.osd.0:Started 2024-03-25T19:41:48.168 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.168 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T19:41:48.168 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T19:41:48.170 INFO:tasks.ceph.osd.1:Started 2024-03-25T19:41:48.170 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.170 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T19:41:48.171 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T19:41:48.173 INFO:tasks.ceph.osd.2:Started 2024-03-25T19:41:48.173 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.173 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T19:41:48.173 DEBUG:teuthology.orchestra.run.smithi130:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T19:41:48.176 INFO:tasks.ceph.osd.3:Started 2024-03-25T19:41:48.176 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.176 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T19:41:48.177 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T19:41:48.207 INFO:tasks.ceph.osd.4:Started 2024-03-25T19:41:48.207 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.208 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T19:41:48.208 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T19:41:48.209 INFO:tasks.ceph.osd.5:Started 2024-03-25T19:41:48.210 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.210 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T19:41:48.210 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T19:41:48.212 INFO:tasks.ceph.osd.6:Started 2024-03-25T19:41:48.212 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T19:41:48.212 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T19:41:48.212 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T19:41:48.214 INFO:tasks.ceph.osd.7:Started 2024-03-25T19:41:48.215 DEBUG:teuthology.orchestra.run.smithi130:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:48.290 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:48.291+0000 7f011fc45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.382+0000 7f0bb5ee4640 10 monclient: get_monmap_and_config 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.382+0000 7f0bb5ee4640 10 monclient: build_initial_monmap 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.382+0000 7f0bb5ee4640 10 monclient: monmap: 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 0 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:41:48.383232+0000 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:41:48.383232+0000 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:48.382 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:48.383 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:48.383 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.383+0000 7f0bb5ee4640 1 Processor -- start 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.383+0000 7f0bb5ee4640 1 -- start start 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.383+0000 7f0bb5ee4640 10 monclient: init 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba80389d0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba8038db0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba8030650 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80c54b0 con 0x7f0ba8030650 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8036420 con 0x7f0ba80389d0 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba8039830 con 0x7f0ba8038db0 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.384+0000 7f0bb5ee4640 10 monclient(hunting): authenticate will time out at 4984.530175s 2024-03-25T19:41:48.385 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.385+0000 7f0baeffd640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f0ba8038db0 0x7f0ba80cadc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:47024/0 (socket says 172.21.15.130:47024) 2024-03-25T19:41:48.386 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.385+0000 7f0baeffd640 1 -- 172.21.15.130:0/445149878 learned_addr learned my addr 172.21.15.130:0/445149878 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:48.386 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:48.386+0000 7f011fc45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466517787 0 0) 0x7f0ba80c54b0 con 0x7f0ba8030650 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 10 monclient(hunting): my global_id is 4170 2024-03-25T19:41:48.490 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.490+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba0003520 con 0x7f0ba8030650 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998889201 0 0) 0x7f0ba8036420 con 0x7f0ba80389d0 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 10 monclient(hunting): my global_id is 4171 2024-03-25T19:41:48.492 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.492+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80c54b0 con 0x7f0ba80389d0 2024-03-25T19:41:48.496 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464488000 0 0) 0x7f0ba0003520 con 0x7f0ba8030650 2024-03-25T19:41:48.496 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 >> v1:172.21.15.130:6790/0 conn(0x7f0ba8038db0 legacy=0x7f0ba80cadc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.496 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 >> v1:172.21.15.163:6789/0 conn(0x7f0ba80389d0 legacy=0x7f0ba8033510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 monclient: found mon.noname-a 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba8035680 con 0x7f0ba8030650 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0bb0075da0 con 0x7f0ba8030650 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bae7fc640 10 monclient: dump: 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.497 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.496+0000 7f0bb5ee4640 5 monclient: authenticate success, global_id 4170 2024-03-25T19:41:48.498 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.497+0000 7f0bb5ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0bb0069d10 con 0x7f0ba8030650 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 1 -- 172.21.15.130:0/445149878 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0bb0076320 con 0x7f0ba8030650 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.500+0000 7f0bae7fc640 10 monclient: dump: 2024-03-25T19:41:48.501 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.501+0000 7f0bb5ee4640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.501+0000 7f0bb5ee4640 4 set_mon_vals no callback set 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.501+0000 7f0bb5ee4640 10 monclient: shutdown 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:48.502 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/445149878 >> v1:172.21.15.130:6789/0 conn(0x7f0ba8030650 legacy=0x7f0ba80d3160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/445149878 shutdown_connections 2024-03-25T19:41:48.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/445149878 >> 172.21.15.130:0/445149878 conn(0x7f0ba8005bf0 msgr2=0x7f0ba8006000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:48.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/445149878 shutdown_connections 2024-03-25T19:41:48.503 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.502+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/445149878 wait complete. 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.503+0000 7f0bb5ee4640 10 monclient: build_initial_monmap 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.503+0000 7f0bb5ee4640 10 monclient: monmap: 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 0 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 0.000000 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:created 0.000000 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 1 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.504+0000 7f0bb5ee4640 1 librados: starting msgr at 2024-03-25T19:41:48.504 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.504+0000 7f0bb5ee4640 1 librados: starting objecter 2024-03-25T19:41:48.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.505+0000 7f0bb5ee4640 1 Processor -- start 2024-03-25T19:41:48.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.505+0000 7f0bb5ee4640 1 -- start start 2024-03-25T19:41:48.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.505+0000 7f0bb5ee4640 1 librados: setting wanted keys 2024-03-25T19:41:48.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.505+0000 7f0bb5ee4640 1 librados: calling monclient init 2024-03-25T19:41:48.505 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.505+0000 7f0bb5ee4640 10 monclient: init 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.506+0000 7f0bb5ee4640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.506+0000 7f0bb5ee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba8030650 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba80389d0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba802f9f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80c7290 con 0x7f0ba8030650 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80c5640 con 0x7f0ba802f9f0 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ba80c58e0 con 0x7f0ba80389d0 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:48.507 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.507+0000 7f0bb5ee4640 10 monclient(hunting): authenticate will time out at 4984.653521s 2024-03-25T19:41:48.508 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.508+0000 7f0baffff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f0ba802f9f0 0x7f0ba8168580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51894/0 (socket says 172.21.15.130:51894) 2024-03-25T19:41:48.508 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.508+0000 7f0baffff640 1 -- 172.21.15.130:0/3703236267 learned_addr learned my addr 172.21.15.130:0/3703236267 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:48.508 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.509+0000 7f0b9bfff640 10 client.?.objecter ms_handle_connect 0x7f0ba802f9f0 2024-03-25T19:41:48.508 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.509+0000 7f0b9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:48.509 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.509+0000 7f0b9bfff640 10 client.?.objecter ms_handle_connect 0x7f0ba8030650 2024-03-25T19:41:48.509 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.509+0000 7f0b9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937336216 0 0) 0x7f0ba80c5640 con 0x7f0ba802f9f0 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 10 monclient(hunting): my global_id is 4174 2024-03-25T19:41:48.512 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.513+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94004eb0 con 0x7f0ba802f9f0 2024-03-25T19:41:48.514 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.515+0000 7f0b9bfff640 10 client.?.objecter ms_handle_connect 0x7f0ba80389d0 2024-03-25T19:41:48.514 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.515+0000 7f0b9bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.520+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1211353703 0 0) 0x7f0b94004eb0 con 0x7f0ba802f9f0 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba80c5640 con 0x7f0ba802f9f0 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0bb00688b0 con 0x7f0ba802f9f0 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:48.520 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.521 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.521+0000 7f0b9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.524+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235433676 0 0) 0x7f0ba80c7290 con 0x7f0ba8030650 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.524+0000 7f0b9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.524+0000 7f0b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.524+0000 7f0b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.525+0000 7f0b9bfff640 10 monclient(hunting): my global_id is 4173 2024-03-25T19:41:48.524 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.525+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b94004eb0 con 0x7f0ba8030650 2024-03-25T19:41:48.527 INFO:tasks.ceph.mgr.y.smithi130.stderr:2024-03-25T19:41:48.528+0000 7f011fc45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T19:41:48.538 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406626135 0 0) 0x7f0ba80c58e0 con 0x7f0ba80389d0 2024-03-25T19:41:48.538 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 10 monclient(hunting): my global_id is 4172 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.539+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ba80c7290 con 0x7f0ba80389d0 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1616207127 0 0) 0x7f0b94004eb0 con 0x7f0ba8030650 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ba80c58e0 con 0x7f0ba8030650 2024-03-25T19:41:48.539 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0ba4001950 con 0x7f0ba8030650 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.540+0000 7f0b9bfff640 10 monclient(hunting): dump: 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:48.540 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.567 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1595615578 0 0) 0x7f0ba80c58e0 con 0x7f0ba8030650 2024-03-25T19:41:48.567 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 >> v1:172.21.15.130:6790/0 conn(0x7f0ba80389d0 legacy=0x7f0ba8159cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.567 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 >> v1:172.21.15.163:6789/0 conn(0x7f0ba802f9f0 legacy=0x7f0ba8168580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 1 monclient: found mon.a 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.567+0000 7f0b9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0b9bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba802fe40 con 0x7f0ba8030650 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0b9bfff640 10 monclient: _finish_auth 0 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0b9bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0b9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 5 monclient: authenticate success, global_id 4173 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 10 monclient: _renew_subs 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ba80c7c90 con 0x7f0ba8030650 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 10 monclient: _renew_subs 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.568 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ba815a3c0 con 0x7f0ba8030650 2024-03-25T19:41:48.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.568+0000 7f0bb5ee4640 1 librados: init done 2024-03-25T19:41:48.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.569+0000 7f0bb5ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:48.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.569+0000 7f0bb5ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:48.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.569+0000 7f0bb5ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.569 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.569+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ba8007b80 con 0x7f0ba8030650 2024-03-25T19:41:48.581 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.582+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ba4003610 con 0x7f0ba8030650 2024-03-25T19:41:48.581 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.582+0000 7f0b9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:48.581 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.582+0000 7f0badffb640 4 set_mon_vals no callback set 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.628+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0ba4003920 con 0x7f0ba8030650 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.628+0000 7f0b9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.628+0000 7f0b9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.628+0000 7f0b9bfff640 10 monclient: dump: 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:epoch 1 2024-03-25T19:41:48.627 INFO:teuthology.orchestra.run.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:48.628 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:41:48.825 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.825+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3009388901 0 0) 0x7f0ba400ea00 con 0x7f0ba8030650 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f0ba40144d0 con 0x7f0ba8030650 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 10 client.4173.objecter ms_dispatch 0x7f0bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0ba4013790 con 0x7f0ba8030650 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:48.829 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.829+0000 7f0b9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:48.950 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.950+0000 7f0bb5ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.950 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.950+0000 7f0bb5ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:48.951 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.950+0000 7f0bb5ee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:48.951 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:48.950+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0ba80c5a70 con 0x7f0ba8030650 2024-03-25T19:41:49.066 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.067+0000 7f0b9bfff640 1 -- 172.21.15.130:0/3703236267 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f0ba4011f70 con 0x7f0ba8030650 2024-03-25T19:41:49.067 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.067+0000 7f0b9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:49.067 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.067+0000 7f0b9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:49.067 INFO:teuthology.orchestra.run.smithi130.stdout: 2024-03-25T19:41:49.067 INFO:teuthology.orchestra.run.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 10 librados: watch_flush enter 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 10 librados: watch_flush exit 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 10 client.4173.objecter successfully canceled tick 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 10 monclient: shutdown 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 >> v1:172.21.15.130:6789/0 conn(0x7f0ba8030650 legacy=0x7f0ba815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:49.068 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 shutdown_connections 2024-03-25T19:41:49.069 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.068+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 >> 172.21.15.130:0/3703236267 conn(0x7f0ba8005bf0 msgr2=0x7f0ba8034080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:49.069 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.069+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 shutdown_connections 2024-03-25T19:41:49.069 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.070+0000 7f0bb5ee4640 1 -- 172.21.15.130:0/3703236267 wait complete. 2024-03-25T19:41:49.069 INFO:teuthology.orchestra.run.smithi130.stderr:2024-03-25T19:41:49.070+0000 7f0bb5ee4640 1 librados: shutdown 2024-03-25T19:41:49.084 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T19:41:49.084 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T19:41:49.709 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:49.710+0000 7fceadbbd640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T19:41:49.745 INFO:tasks.ceph.mgr.x.smithi163.stderr:2024-03-25T19:41:49.746+0000 7fceadbbd640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T19:41:49.891 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.034+0000 7f141ea9a640 10 monclient: get_monmap_and_config 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.034+0000 7f141ea9a640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.034+0000 7f141ea9a640 10 monclient: monmap: 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:41:50.035502+0000 2024-03-25T19:41:50.034 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:41:50.035502+0000 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:50.035 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.035+0000 7f141ea9a640 1 Processor -- start 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.035+0000 7f141ea9a640 1 -- start start 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.035+0000 7f141ea9a640 10 monclient: init 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.036+0000 7f141ea9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.036+0000 7f141ea9a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.036+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-c con 0x7f14180b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:50.036 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.036+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-a con 0x7f14180b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:50.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-b con 0x7f141814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:50.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180af940 con 0x7f14180b1730 2024-03-25T19:41:50.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180b0a70 con 0x7f14180b4d60 2024-03-25T19:41:50.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14180b0d50 con 0x7f141814be90 2024-03-25T19:41:50.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f141ea9a640 10 monclient(hunting): authenticate will time out at 4986.183114s 2024-03-25T19:41:50.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f1417fff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f14180b1730 0x7f14180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:39048/0 (socket says 172.21.15.130:39048) 2024-03-25T19:41:50.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.037+0000 7f1417fff640 1 -- 172.21.15.130:0/1163348485 learned_addr learned my addr 172.21.15.130:0/1163348485 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414219621 0 0) 0x7f14180af940 con 0x7f14180b1730 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): my global_id is 4176 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1408003520 con 0x7f14180b1730 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173627941 0 0) 0x7f14180b0d50 con 0x7f141814be90 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): my global_id is 4175 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14180af940 con 0x7f141814be90 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597249705 0 0) 0x7f14180b0a70 con 0x7f14180b4d60 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.042 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.042 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 10 monclient(hunting): my global_id is 4177 2024-03-25T19:41:50.042 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.041+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14180b0d50 con 0x7f14180b4d60 2024-03-25T19:41:50.045 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 910404030 0 0) 0x7f14180af940 con 0x7f141814be90 2024-03-25T19:41:50.045 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 >> v1:172.21.15.163:6789/0 conn(0x7f14180b4d60 legacy=0x7f141813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.045 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 >> v1:172.21.15.130:6789/0 conn(0x7f14180b1730 legacy=0x7f14180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 1 monclient: found mon.noname-b 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14180b1030 con 0x7f141814be90 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f14177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f141ea9a640 5 monclient: authenticate success, global_id 4175 2024-03-25T19:41:50.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.046+0000 7f141ea9a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:50.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f14100019b0 con 0x7f141814be90 2024-03-25T19:41:50.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:50.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: dump: 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1410001c00 con 0x7f141814be90 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 1 -- 172.21.15.130:0/1163348485 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1410001fd0 con 0x7f141814be90 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.049 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f14177fe640 10 monclient: dump: 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f141ea9a640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:50.050 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.049+0000 7f141ea9a640 4 set_mon_vals no callback set 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 10 monclient: shutdown 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 1 -- 172.21.15.130:0/1163348485 >> v1:172.21.15.130:6790/0 conn(0x7f141814be90 legacy=0x7f141814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 1 -- 172.21.15.130:0/1163348485 shutdown_connections 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 1 -- 172.21.15.130:0/1163348485 >> 172.21.15.130:0/1163348485 conn(0x7f14180617a0 msgr2=0x7f1418061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 1 -- 172.21.15.130:0/1163348485 shutdown_connections 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.050+0000 7f141ea9a640 1 -- 172.21.15.130:0/1163348485 wait complete. 2024-03-25T19:41:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.051+0000 7f141ea9a640 10 monclient: build_initial_monmap 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.051+0000 7f141ea9a640 10 monclient: monmap: 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 librados: starting msgr at 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 librados: starting objecter 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 Processor -- start 2024-03-25T19:41:50.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 -- start start 2024-03-25T19:41:50.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 librados: setting wanted keys 2024-03-25T19:41:50.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 1 librados: calling monclient init 2024-03-25T19:41:50.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.052+0000 7f141ea9a640 10 monclient: init 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.053+0000 7f141ea9a640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.053+0000 7f141ea9a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.053+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-b con 0x7f14180b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-c con 0x7f14180b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 10 monclient(hunting): picked mon.noname-a con 0x7f141814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14181d7820 con 0x7f141814be90 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14181d5360 con 0x7f14180b1730 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f141813d0f0 con 0x7f14180b4d60 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f141ea9a640 10 monclient(hunting): authenticate will time out at 4986.200156s 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f1417fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f14180b4d60 0x7f14181d7cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:47050/0 (socket says 172.21.15.130:47050) 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.054+0000 7f1417fff640 1 -- 172.21.15.130:0/547724874 learned_addr learned my addr 172.21.15.130:0/547724874 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter ms_handle_connect 0x7f14180b4d60 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter ms_handle_connect 0x7f141814be90 2024-03-25T19:41:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter ms_handle_connect 0x7f14180b1730 2024-03-25T19:41:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.055+0000 7f14157fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950080638 0 0) 0x7f141813d0f0 con 0x7f14180b4d60 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): my global_id is 4178 2024-03-25T19:41:50.058 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13fc004eb0 con 0x7f14180b4d60 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3442531197 0 0) 0x7f14181d7820 con 0x7f141814be90 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): my global_id is 4179 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141813d0f0 con 0x7f141814be90 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 735687577 0 0) 0x7f14181d5360 con 0x7f14180b1730 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _negotiate 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 10 monclient(hunting): my global_id is 4180 2024-03-25T19:41:50.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.058+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14181d7820 con 0x7f14180b1730 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.065+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3270502034 0 0) 0x7f141813d0f0 con 0x7f141814be90 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.065+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14181d5360 con 0x7f141814be90 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3357669076 0 0) 0x7f14181d7820 con 0x7f14180b1730 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f141813d0f0 con 0x7f14180b1730 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f140c001830 con 0x7f14180b1730 2024-03-25T19:41:50.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): dump: 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3707821624 0 0) 0x7f13fc004eb0 con 0x7f14180b4d60 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14181d7820 con 0x7f14180b4d60 2024-03-25T19:41:50.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1410001f10 con 0x7f141814be90 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): dump: 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1404001970 con 0x7f14180b4d60 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.066+0000 7f14157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.067+0000 7f14157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.067+0000 7f14157fa640 10 monclient(hunting): dump: 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.069 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.071+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3773875553 0 0) 0x7f141813d0f0 con 0x7f14180b1730 2024-03-25T19:41:50.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 >> v1:172.21.15.130:6790/0 conn(0x7f14180b4d60 legacy=0x7f14181d7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 >> v1:172.21.15.130:6789/0 conn(0x7f141814be90 legacy=0x7f14181e3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 10 monclient: _finish_hunting 0 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 1 monclient: found mon.b 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1418133630 con 0x7f14180b1730 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 10 monclient: _finish_auth 0 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 10 monclient: _check_auth_tickets 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f14157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f141ea9a640 5 monclient: authenticate success, global_id 4180 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f141ea9a640 10 monclient: _renew_subs 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f141ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1418141030 con 0x7f14180b1730 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.072+0000 7f141ea9a640 10 client.4180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.073+0000 7f141ea9a640 10 monclient: _renew_subs 2024-03-25T19:41:50.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.073+0000 7f141ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.073+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1418140ce0 con 0x7f14180b1730 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.073+0000 7f141ea9a640 1 librados: init done 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.074+0000 7f141ea9a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.074+0000 7f141ea9a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.074+0000 7f141ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:50.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.074+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14181d79d0 con 0x7f14180b1730 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f140c001d40 con 0x7f14180b1730 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f140c003650 con 0x7f14180b1730 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f1416ffd640 4 set_mon_vals no callback set 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.101+0000 7f14157fa640 10 monclient: dump: 2024-03-25T19:41:50.101 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:50.102 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:50.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.102+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 3585832389 0 0) 0x7f140c019670 con 0x7f14180b1730 2024-03-25T19:41:50.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.103+0000 7f1417fff640 10 monclient: get_auth_request con 0x7f13fc06ded0 auth_method 0 2024-03-25T19:41:50.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.103+0000 7f14157fa640 10 client.4180.objecter ms_handle_connect 0x7f13fc06ded0 2024-03-25T19:41:50.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.106+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f140c053350 con 0x7f14180b1730 2024-03-25T19:41:50.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.106+0000 7f14157fa640 10 client.4180.objecter ms_dispatch 0x7f1418000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:50.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.106+0000 7f14157fa640 3 client.4180.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:50.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.106+0000 7f14157fa640 3 client.4180.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:50.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.106+0000 7f14157fa640 20 client.4180.objecter dump_active .. 0 homeless 2024-03-25T19:41:50.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.513+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f140c04ee60 con 0x7f14180b1730 2024-03-25T19:41:50.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.513+0000 7f14157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:50.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.513+0000 7f14157fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:50.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.731+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f140c04e910 con 0x7f14180b1730 2024-03-25T19:41:50.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.876+0000 7f141ea9a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.876+0000 7f141ea9a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.876+0000 7f141ea9a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:41:50.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.876+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14181d79d0 con 0x7f14180b1730 2024-03-25T19:41:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.949+0000 7f14157fa640 1 -- 172.21.15.130:0/547724874 <== mon.1 v1:172.21.15.163:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f14181d79d0 con 0x7f14180b1730 2024-03-25T19:41:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.949+0000 7f14157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.949+0000 7f14157fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:50.949 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:41:50.949 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.951+0000 7f141ea9a640 10 librados: watch_flush enter 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 10 librados: watch_flush exit 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 20 client.4180.objecter shutdown clearing up homeless session... 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 10 client.4180.objecter successfully canceled tick 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 >> v1:172.21.15.163:6800/1217380380 conn(0x7f13fc06ded0 legacy=0x7f13fc070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 10 monclient: shutdown 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 >> v1:172.21.15.163:6789/0 conn(0x7f14180b1730 legacy=0x7f1418132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 shutdown_connections 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 >> 172.21.15.130:0/547724874 conn(0x7f14180617a0 msgr2=0x7f14180b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 shutdown_connections 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 -- 172.21.15.130:0/547724874 wait complete. 2024-03-25T19:41:50.952 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:50.952+0000 7f141ea9a640 1 librados: shutdown 2024-03-25T19:41:50.965 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:55.059 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:06.768 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.069 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:06.780 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.122 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:06.832 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.155 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:06.866 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.199 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:06.921 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.258 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:06.980 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.308 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:07.042 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.356 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:07.079 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.375 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:07.110 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.413 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:07.136 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.511 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:07.249 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.570 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:07.308 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.571 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:07.306 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.585 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:07.324 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.631 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:07.366 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.644 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:07.384 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:55.680 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-03-25T19:41:55.675+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:55.703 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-03-25T19:41:55.698+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:55.840 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-03-25T19:41:55.835+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:55.916 INFO:tasks.ceph.osd.2.smithi130.stderr:2024-03-25T19:41:55.911+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:55.996 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-03-25T19:41:55.991+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:56.144 INFO:tasks.ceph.osd.3.smithi130.stderr:2024-03-25T19:41:56.138+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:56.163 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:07.873 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.164 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:07.875 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.199 INFO:tasks.ceph.osd.0.smithi130.stderr:2024-03-25T19:41:56.192+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:56.207 INFO:tasks.ceph.osd.1.smithi130.stderr:2024-03-25T19:41:56.203+0000 5dc3480 -1 Falling back to public interface 2024-03-25T19:41:56.310 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:08.032 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.375 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:08.109 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.458 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:08.182 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.530 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:08.240 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.533 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:08.244 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.667 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:08.406 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.668 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:08.402 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.674 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:08.396 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.680 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:08.419 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.757 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:08.491 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.820 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:08.544 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.846 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:08.556 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.896 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:08.607 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:56.984 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:08.706 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.008 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:08.743 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.056 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:08.794 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.063 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:08.803 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.075 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:08.809 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.113 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:08.823 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.148 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:08.871 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.166 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:08.877 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.249 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:08.970 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.322 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:09.057 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.345 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:09.079 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.362 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:09.101 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.376 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:09.087 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.379 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:09.117 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.413 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:09.136 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.454 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:09.165 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.518 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:09.239 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.587 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:09.321 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.614 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:09.349 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.632 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:09.372 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.641 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:09.351 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.650 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:09.388 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.677 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:09.400 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.743 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:09.453 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.773 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:41:57.785 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:09.507 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.862+0000 7fd34fce6640 10 monclient: get_monmap_and_config 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.862+0000 7fd34fce6640 10 monclient: build_initial_monmap 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.862+0000 7fd34fce6640 10 monclient: monmap: 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.862 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:41:57.863611+0000 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:41:57.863611+0000 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:41:57.863 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.863+0000 7fd34fce6640 1 Processor -- start 2024-03-25T19:41:57.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.863+0000 7fd34fce6640 1 -- start start 2024-03-25T19:41:57.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.863+0000 7fd34fce6640 10 monclient: init 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-a con 0x7fd34806c380 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-c con 0x7fd34806d670 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-b con 0x7fd3480af8f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34806c9a0 con 0x7fd34806c380 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480ada90 con 0x7fd34806d670 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3480b0d10 con 0x7fd3480af8f0 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34fce6640 10 monclient(hunting): authenticate will time out at 4994.009818s 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34e283640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fd3480af8f0 0x7fd3480aedf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:34804/0 (socket says 172.21.15.130:34804) 2024-03-25T19:41:57.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.864+0000 7fd34e283640 1 -- 172.21.15.130:0/2826170762 learned_addr learned my addr 172.21.15.130:0/2826170762 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:57.866 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:09.598 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763277408 0 0) 0x7fd3480ada90 con 0x7fd34806d670 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): my global_id is 4207 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd334003520 con 0x7fd34806d670 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080135953 0 0) 0x7fd34806c9a0 con 0x7fd34806c380 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): my global_id is 4206 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3480ada90 con 0x7fd34806c380 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3452810083 0 0) 0x7fd3480b0d10 con 0x7fd3480af8f0 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 10 monclient(hunting): my global_id is 4205 2024-03-25T19:41:57.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.867+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34806c9a0 con 0x7fd3480af8f0 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 67460651 0 0) 0x7fd334003520 con 0x7fd34806d670 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 >> v1:172.21.15.130:6790/0 conn(0x7fd3480af8f0 legacy=0x7fd3480aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 >> v1:172.21.15.130:6789/0 conn(0x7fd34806c380 legacy=0x7fd3480b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: _finish_hunting 0 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 monclient: found mon.noname-c 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3480add00 con 0x7fd34806d670 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: _finish_auth 0 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: _check_auth_tickets 2024-03-25T19:41:57.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd338001690 con 0x7fd34806d670 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34ca80640 10 monclient: dump: 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.871 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34fce6640 5 monclient: authenticate success, global_id 4207 2024-03-25T19:41:57.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.870+0000 7fd34fce6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:41:57.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3380017f0 con 0x7fd34806d670 2024-03-25T19:41:57.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:57.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 1 -- 172.21.15.130:0/2826170762 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd338001b80 con 0x7fd34806d670 2024-03-25T19:41:57.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34ca80640 10 monclient: dump: 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34fce6640 10 monclient: get_monmap_and_config success 2024-03-25T19:41:57.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34fce6640 4 set_mon_vals no callback set 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.872+0000 7fd34fce6640 10 monclient: shutdown 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 1 -- 172.21.15.130:0/2826170762 >> v1:172.21.15.163:6789/0 conn(0x7fd34806d670 legacy=0x7fd3480ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 1 -- 172.21.15.130:0/2826170762 shutdown_connections 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 1 -- 172.21.15.130:0/2826170762 >> 172.21.15.130:0/2826170762 conn(0x7fd3480b5c00 msgr2=0x7fd3480b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 1 -- 172.21.15.130:0/2826170762 shutdown_connections 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 1 -- 172.21.15.130:0/2826170762 wait complete. 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 10 monclient: build_initial_monmap 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.873+0000 7fd34fce6640 10 monclient: monmap: 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:41:57.875 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 librados: starting msgr at 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 librados: starting objecter 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 Processor -- start 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 -- start start 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 librados: setting wanted keys 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 librados: calling monclient init 2024-03-25T19:41:57.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient: init 2024-03-25T19:41:57.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient: _reopen_session rank -1 2024-03-25T19:41:57.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:41:57.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-b con 0x7fd34806c380 addr v1:172.21.15.163:6789/0 2024-03-25T19:41:57.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-a con 0x7fd34806d670 addr v1:172.21.15.130:6789/0 2024-03-25T19:41:57.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient(hunting): picked mon.noname-c con 0x7fd3480af8f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34814e490 con 0x7fd34806d670 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3481d7ad0 con 0x7fd34806c380 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3481d4d80 con 0x7fd3480af8f0 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient(hunting): _renew_subs 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.874+0000 7fd34fce6640 10 monclient(hunting): authenticate will time out at 4994.020440s 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd34e283640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fd3480af8f0 0x7fd3481e1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:34820/0 (socket says 172.21.15.130:34820) 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd34e283640 1 -- 172.21.15.130:0/235165384 learned_addr learned my addr 172.21.15.130:0/235165384 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter ms_handle_connect 0x7fd3480af8f0 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter ms_handle_connect 0x7fd34806d670 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter ms_handle_connect 0x7fd34806c380 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.875+0000 7fd33e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050386116 0 0) 0x7fd34814e490 con 0x7fd34806d670 2024-03-25T19:41:57.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c004eb0 con 0x7fd34806d670 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436347058 0 0) 0x7fd3481d4d80 con 0x7fd3480af8f0 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): my global_id is 4208 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd34814e490 con 0x7fd3480af8f0 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2195445960 0 0) 0x7fd3481d7ad0 con 0x7fd34806c380 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:41:57.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:41:57.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 10 monclient(hunting): my global_id is 4210 2024-03-25T19:41:57.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.877+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3481d4d80 con 0x7fd34806c380 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2048370912 0 0) 0x7fd32c004eb0 con 0x7fd34806d670 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3481d7ad0 con 0x7fd34806d670 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd338001b00 con 0x7fd34806d670 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.882 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408370608 0 0) 0x7fd3481d4d80 con 0x7fd34806c380 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd32c004eb0 con 0x7fd34806c380 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd344001c70 con 0x7fd34806c380 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.883 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1869154754 0 0) 0x7fd34814e490 con 0x7fd3480af8f0 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3481d4d80 con 0x7fd3480af8f0 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd340001790 con 0x7fd3480af8f0 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.882+0000 7fd33e7fc640 10 monclient(hunting): dump: 2024-03-25T19:41:57.884 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.885 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1698147393 0 0) 0x7fd3481d7ad0 con 0x7fd34806d670 2024-03-25T19:41:57.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 >> v1:172.21.15.130:6790/0 conn(0x7fd3480af8f0 legacy=0x7fd3481e1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 >> v1:172.21.15.163:6789/0 conn(0x7fd34806c380 legacy=0x7fd3481d72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 1 monclient: found mon.a 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3481d4f10 con 0x7fd34806d670 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd33e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 5 monclient: authenticate success, global_id 4209 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 10 monclient: _renew_subs 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34814eb00 con 0x7fd34806d670 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:41:57.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 10 monclient: _renew_subs 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3481d81d0 con 0x7fd34806d670 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.885+0000 7fd34fce6640 1 librados: init done 2024-03-25T19:41:57.888 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:09.619 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.886+0000 7fd34fce6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.886+0000 7fd34fce6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:57.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.886+0000 7fd34fce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:57.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.886+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3480627b0 con 0x7fd34806d670 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.900+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3380010e0 con 0x7fd34806d670 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.900+0000 7fd33e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.901+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd338003a40 con 0x7fd34806d670 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.901+0000 7fd33e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.901+0000 7fd33e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.901+0000 7fd33e7fc640 10 monclient: dump: 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:41:57.900 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:41:57.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.901+0000 7fd33ffff640 4 set_mon_vals no callback set 2024-03-25T19:41:57.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.902+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fd338019a10 con 0x7fd34806d670 2024-03-25T19:41:57.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.902+0000 7fd34da82640 10 monclient: get_auth_request con 0x7fd32c06e060 auth_method 0 2024-03-25T19:41:57.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.903+0000 7fd33e7fc640 10 client.4209.objecter ms_handle_connect 0x7fd32c06e060 2024-03-25T19:41:57.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.904+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fd3380535a0 con 0x7fd34806d670 2024-03-25T19:41:57.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.904+0000 7fd33e7fc640 10 client.4209.objecter ms_dispatch 0x7fd348000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:41:57.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.904+0000 7fd33e7fc640 3 client.4209.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:41:57.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.904+0000 7fd33e7fc640 3 client.4209.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:41:57.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:57.904+0000 7fd33e7fc640 20 client.4209.objecter dump_active .. 0 homeless 2024-03-25T19:41:57.906 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:09.616 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.906 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:09.645 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.926 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:09.664 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:57.941 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:09.664 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.034 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:09.745 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.054 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:09.776 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.130 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:09.864 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.131+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd33804eee0 con 0x7fd34806d670 2024-03-25T19:41:58.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.131+0000 7fd33e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:41:58.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.131+0000 7fd33e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:41:58.157 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:09.892 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.174 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:09.884 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.180 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:09.919 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.197 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:09.935 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.207 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:09.930 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.307+0000 7fd34fce6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:58.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.307+0000 7fd34fce6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:58.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.307+0000 7fd34fce6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:41:58.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.307+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3481d7c60 con 0x7fd34806d670 2024-03-25T19:41:58.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.321+0000 7fd33e7fc640 1 -- 172.21.15.130:0/235165384 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fd338050fe0 con 0x7fd34806d670 2024-03-25T19:41:58.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.321+0000 7fd33e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:41:58.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.321+0000 7fd33e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:41:58.321 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:41:58.321 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 10 librados: watch_flush enter 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 10 librados: watch_flush exit 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 20 client.4209.objecter shutdown clearing up homeless session... 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 10 client.4209.objecter successfully canceled tick 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 >> v1:172.21.15.163:6800/1217380380 conn(0x7fd32c06e060 legacy=0x7fd32c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 10 monclient: shutdown 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 >> v1:172.21.15.130:6789/0 conn(0x7fd34806d670 legacy=0x7fd34814dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:41:58.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 shutdown_connections 2024-03-25T19:41:58.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 >> 172.21.15.130:0/235165384 conn(0x7fd3480b5c00 msgr2=0x7fd3480b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:41:58.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 shutdown_connections 2024-03-25T19:41:58.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 -- 172.21.15.130:0/235165384 wait complete. 2024-03-25T19:41:58.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:41:58.323+0000 7fd34fce6640 1 librados: shutdown 2024-03-25T19:41:58.324 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:10.045 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.327 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:10.038 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.332 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:41:58.421 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:10.155 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.426 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:10.161 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.446 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:10.156 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.456 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:10.196 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.467 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:10.205 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.474 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:10.197 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.588 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:10.309 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.619 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:10.330 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.700 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:10.434 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.704 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:10.439 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.730 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:10.469 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.738 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:10.477 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.750 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:10.474 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.915 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:10.625 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.967 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:10.701 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.969 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:10.679 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:58.995 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:10.735 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.015 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:10.754 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.089 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:10.811 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.151 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:10.873 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.180 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:10.915 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.188 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:10.899 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.228 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:10.951 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.232 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:10.967 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.243 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:10.953 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.282 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:11.005 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.449 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:11.183 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.493 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:11.231 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.516 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:11.255 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.528 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:11.263 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.548 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:11.286 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:41:59.570 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:11.310 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.584 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:16.318 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.771 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:16.494 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.826 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:16.536 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.845 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:16.556 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.850 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:16.589 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.867 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:16.588 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:04.905 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:16.639 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:05.139 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:05.140 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:16.880 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:05.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.275+0000 7fba09c74640 10 monclient: get_monmap_and_config 2024-03-25T19:42:05.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.275+0000 7fba09c74640 10 monclient: build_initial_monmap 2024-03-25T19:42:05.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.275+0000 7fba09c74640 10 monclient: monmap: 2024-03-25T19:42:05.274 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:05.274 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:05.275901+0000 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:05.275901+0000 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:05.275 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 1 Processor -- start 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 1 -- start start 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 10 monclient: init 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.276+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-a con 0x7fba040b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-b con 0x7fba040b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-c con 0x7fba0414be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba040af940 con 0x7fba040b4d60 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba040b0a70 con 0x7fba0414be90 2024-03-25T19:42:05.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba040b0d50 con 0x7fba040b1730 2024-03-25T19:42:05.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:05.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba09c74640 10 monclient(hunting): authenticate will time out at 5001.422699s 2024-03-25T19:42:05.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba02ffd640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fba040b1730 0x7fba040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:34834/0 (socket says 172.21.15.130:34834) 2024-03-25T19:42:05.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.277+0000 7fba02ffd640 1 -- 172.21.15.130:0/120516369 learned_addr learned my addr 172.21.15.130:0/120516369 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3037134341 0 0) 0x7fba040af940 con 0x7fba040b4d60 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 10 monclient(hunting): my global_id is 4212 2024-03-25T19:42:05.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.280+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec003520 con 0x7fba040b4d60 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299525392 0 0) 0x7fba040b0d50 con 0x7fba040b1730 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): my global_id is 4211 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba040af940 con 0x7fba040b1730 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218715298 0 0) 0x7fba040b0a70 con 0x7fba0414be90 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 10 monclient(hunting): my global_id is 4213 2024-03-25T19:42:05.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.281+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9ec0039d0 con 0x7fba0414be90 2024-03-25T19:42:05.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1095057796 0 0) 0x7fb9ec0039d0 con 0x7fba0414be90 2024-03-25T19:42:05.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 >> v1:172.21.15.130:6790/0 conn(0x7fba040b1730 legacy=0x7fba040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 >> v1:172.21.15.130:6789/0 conn(0x7fba040b4d60 legacy=0x7fba0413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 monclient: found mon.noname-c 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba040b1030 con 0x7fba0414be90 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba09c74640 5 monclient: authenticate success, global_id 4213 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba09c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9fc001930 con 0x7fba0414be90 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba027fc640 10 monclient: dump: 2024-03-25T19:42:05.286 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.286+0000 7fba09c74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9fc001b10 con 0x7fba0414be90 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 1 -- 172.21.15.130:0/120516369 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9fc001930 con 0x7fba0414be90 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba027fc640 10 monclient: dump: 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.288 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.289 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.289 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.289 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.289 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba09c74640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:05.289 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.288+0000 7fba09c74640 4 set_mon_vals no callback set 2024-03-25T19:42:05.290 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 10 monclient: shutdown 2024-03-25T19:42:05.290 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.290 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 1 -- 172.21.15.130:0/120516369 >> v1:172.21.15.163:6789/0 conn(0x7fba0414be90 legacy=0x7fba0414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.290 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 1 -- 172.21.15.130:0/120516369 shutdown_connections 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 1 -- 172.21.15.130:0/120516369 >> 172.21.15.130:0/120516369 conn(0x7fba040617a0 msgr2=0x7fba04061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 1 -- 172.21.15.130:0/120516369 shutdown_connections 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.289+0000 7fba09c74640 1 -- 172.21.15.130:0/120516369 wait complete. 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.290+0000 7fba09c74640 10 monclient: build_initial_monmap 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.290+0000 7fba09c74640 10 monclient: monmap: 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:05.291 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.290+0000 7fba09c74640 1 librados: starting msgr at 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.290+0000 7fba09c74640 1 librados: starting objecter 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.290+0000 7fba09c74640 1 Processor -- start 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 1 -- start start 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 1 librados: setting wanted keys 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 1 librados: calling monclient init 2024-03-25T19:42:05.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient: init 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-c con 0x7fba040b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-a con 0x7fba040b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 10 monclient(hunting): picked mon.noname-b con 0x7fba0414be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba041dc580 con 0x7fba040b4d60 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.291+0000 7fba09c74640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba04143300 con 0x7fba0414be90 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fba09c74640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba041dc110 con 0x7fba040b1730 2024-03-25T19:42:05.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fba09c74640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fba09c74640 10 monclient(hunting): authenticate will time out at 5001.437651s 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fba037fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fba040b1730 0x7fba0413aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:34840/0 (socket says 172.21.15.130:34840) 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fba037fe640 1 -- 172.21.15.130:0/3987280643 learned_addr learned my addr 172.21.15.130:0/3987280643 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fb9ebfff640 10 client.?.objecter ms_handle_connect 0x7fba040b1730 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.292+0000 7fb9ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.293+0000 7fb9ebfff640 10 client.?.objecter ms_handle_connect 0x7fba040b4d60 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.293+0000 7fb9ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.293+0000 7fb9ebfff640 10 client.?.objecter ms_handle_connect 0x7fba0414be90 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.293+0000 7fb9ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349418246 0 0) 0x7fba04143300 con 0x7fba0414be90 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4216 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f0004eb0 con 0x7fba0414be90 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946801442 0 0) 0x7fba041dc580 con 0x7fba040b4d60 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.295+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4215 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba04143300 con 0x7fba040b4d60 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345364199 0 0) 0x7fba041dc110 con 0x7fba040b1730 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4214 2024-03-25T19:42:05.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.296+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba041dc580 con 0x7fba040b1730 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 506029853 0 0) 0x7fb9f0004eb0 con 0x7fba0414be90 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba041dc110 con 0x7fba0414be90 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9fc001930 con 0x7fba0414be90 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 10 monclient(hunting): dump: 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.300 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.301 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.300+0000 7fb9ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:05.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3426648193 0 0) 0x7fba041dc580 con 0x7fba040b1730 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f0004eb0 con 0x7fba040b1730 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9f40018d0 con 0x7fba040b1730 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): dump: 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.306 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2090903225 0 0) 0x7fba04143300 con 0x7fba040b4d60 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba041dc580 con 0x7fba040b4d60 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9f80018b0 con 0x7fba040b4d60 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.303+0000 7fb9ebfff640 10 monclient(hunting): dump: 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.307 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2911817562 0 0) 0x7fba041dc110 con 0x7fba0414be90 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 >> v1:172.21.15.130:6790/0 conn(0x7fba040b1730 legacy=0x7fba0413aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 >> v1:172.21.15.130:6789/0 conn(0x7fba040b4d60 legacy=0x7fba041dca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 1 monclient: found mon.b 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba041dc2a0 con 0x7fba0414be90 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 10 monclient: _finish_auth 0 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fb9ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 5 monclient: authenticate success, global_id 4216 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 10 monclient: _renew_subs 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba04139160 con 0x7fba0414be90 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 10 monclient: _renew_subs 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba04134bb0 con 0x7fba0414be90 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.304+0000 7fba09c74640 1 librados: init done 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.305+0000 7fba09c74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.305+0000 7fba09c74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.305+0000 7fba09c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.305+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba040afbd0 con 0x7fba0414be90 2024-03-25T19:42:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9fc003c20 con 0x7fba0414be90 2024-03-25T19:42:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb9fc003f30 con 0x7fba0414be90 2024-03-25T19:42:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fb9ebfff640 10 monclient: dump: 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.326+0000 7fba01ffb640 4 set_mon_vals no callback set 2024-03-25T19:42:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.328+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fb9fc019f00 con 0x7fba0414be90 2024-03-25T19:42:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.328+0000 7fba037fe640 10 monclient: get_auth_request con 0x7fb9f006dfe0 auth_method 0 2024-03-25T19:42:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.329+0000 7fb9ebfff640 10 client.4216.objecter ms_handle_connect 0x7fb9f006dfe0 2024-03-25T19:42:05.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.331+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fb9fc053930 con 0x7fba0414be90 2024-03-25T19:42:05.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.331+0000 7fb9ebfff640 10 client.4216.objecter ms_dispatch 0x7fba04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:05.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.331+0000 7fb9ebfff640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:05.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.331+0000 7fb9ebfff640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:05.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.331+0000 7fb9ebfff640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T19:42:05.607 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.608+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9fc04f270 con 0x7fba0414be90 2024-03-25T19:42:05.607 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.608+0000 7fb9ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:05.607 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.608+0000 7fb9ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:05.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.891+0000 7fba09c74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.891+0000 7fba09c74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.891+0000 7fba09c74640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:05.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.891+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba041436c0 con 0x7fba0414be90 2024-03-25T19:42:05.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.914+0000 7fb9ebfff640 1 -- 172.21.15.130:0/3987280643 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fb9fc051370 con 0x7fba0414be90 2024-03-25T19:42:05.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.914+0000 7fb9ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:05.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.914+0000 7fb9ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:05.914 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:05.914 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 10 librados: watch_flush enter 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 10 librados: watch_flush exit 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 10 client.4216.objecter successfully canceled tick 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 >> v1:172.21.15.163:6800/1217380380 conn(0x7fb9f006dfe0 legacy=0x7fb9f00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 10 monclient: shutdown 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 >> v1:172.21.15.163:6789/0 conn(0x7fba0414be90 legacy=0x7fba041344a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:05.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 shutdown_connections 2024-03-25T19:42:05.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 >> 172.21.15.130:0/3987280643 conn(0x7fba040617a0 msgr2=0x7fba0413c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:05.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 shutdown_connections 2024-03-25T19:42:05.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 -- 172.21.15.130:0/3987280643 wait complete. 2024-03-25T19:42:05.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:05.916+0000 7fba09c74640 1 librados: shutdown 2024-03-25T19:42:05.928 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:09.960 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:21.670 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:09.967 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:21.677 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.245 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:21.966 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.249 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:21.971 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.320 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:22.043 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.323 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:22.047 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.402 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:22.113 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.408 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:22.119 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.780 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:22.519 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.784 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:22.523 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.809 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:22.544 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:10.813 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:22.548 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:11.035 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:22.774 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:11.039 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:22.779 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:11.070 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:22.805 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:11.074 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:22.809 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:12.736 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:12.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.870+0000 7fcd3f4d6640 10 monclient: get_monmap_and_config 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.870+0000 7fcd3f4d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.870+0000 7fcd3f4d6640 10 monclient: monmap: 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:12.871277+0000 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:12.871277+0000 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:12.870 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.871+0000 7fcd3f4d6640 1 Processor -- start 2024-03-25T19:42:12.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.871+0000 7fcd3f4d6640 1 -- start start 2024-03-25T19:42:12.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.871+0000 7fcd3f4d6640 10 monclient: init 2024-03-25T19:42:12.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:12.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcd380b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:12.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcd380b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:12.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcd3814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.872+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380af940 con 0x7fcd380b4d60 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380b0a70 con 0x7fcd3814be90 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380b0d50 con 0x7fcd380b1730 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3f4d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3f4d6640 10 monclient(hunting): authenticate will time out at 5009.018649s 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3ca71640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fcd380b1730 0x7fcd380b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:35728/0 (socket says 172.21.15.130:35728) 2024-03-25T19:42:12.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.873+0000 7fcd3ca71640 1 -- 172.21.15.130:0/3282346919 learned_addr learned my addr 172.21.15.130:0/3282346919 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1961009761 0 0) 0x7fcd380b0d50 con 0x7fcd380b1730 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): my global_id is 4217 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd24003520 con 0x7fcd380b1730 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630155549 0 0) 0x7fcd380b0a70 con 0x7fcd3814be90 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): my global_id is 4219 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd380b0d50 con 0x7fcd3814be90 2024-03-25T19:42:12.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355469443 0 0) 0x7fcd380af940 con 0x7fcd380b4d60 2024-03-25T19:42:12.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 10 monclient(hunting): my global_id is 4218 2024-03-25T19:42:12.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.876+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd380b0a70 con 0x7fcd380b4d60 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4276215488 0 0) 0x7fcd380b0d50 con 0x7fcd3814be90 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 >> v1:172.21.15.130:6790/0 conn(0x7fcd380b1730 legacy=0x7fcd380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 >> v1:172.21.15.130:6789/0 conn(0x7fcd380b4d60 legacy=0x7fcd3813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 monclient: found mon.noname-c 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd380b1030 con 0x7fcd3814be90 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd3f4d6640 5 monclient: authenticate success, global_id 4219 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd3f4d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcd300019b0 con 0x7fcd3814be90 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd2ffff640 10 monclient: dump: 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:12.882 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:12.883 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:12.883 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.881+0000 7fcd3f4d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd30001c00 con 0x7fcd3814be90 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd3f4d6640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd3f4d6640 4 set_mon_vals no callback set 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 1 -- 172.21.15.130:0/3282346919 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcd30001f90 con 0x7fcd3814be90 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.883+0000 7fcd2ffff640 10 monclient: dump: 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 10 monclient: shutdown 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:12.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/3282346919 >> v1:172.21.15.163:6789/0 conn(0x7fcd3814be90 legacy=0x7fcd3814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/3282346919 shutdown_connections 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/3282346919 >> 172.21.15.130:0/3282346919 conn(0x7fcd380617a0 msgr2=0x7fcd38061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/3282346919 shutdown_connections 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.884+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/3282346919 wait complete. 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient: build_initial_monmap 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient: monmap: 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:12.887 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 librados: starting msgr at 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 librados: starting objecter 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 Processor -- start 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 -- start start 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 librados: setting wanted keys 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 1 librados: calling monclient init 2024-03-25T19:42:12.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient: init 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.885+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-a con 0x7fcd380b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-b con 0x7fcd380b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 10 monclient(hunting): picked mon.noname-c con 0x7fcd3814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd381d7b00 con 0x7fcd380b1730 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd38142830 con 0x7fcd380b4d60 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd381d7620 con 0x7fcd3814be90 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3f4d6640 10 monclient(hunting): authenticate will time out at 5009.031769s 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3d272640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fcd380b1730 0x7fcd38140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:53936/0 (socket says 172.21.15.130:53936) 2024-03-25T19:42:12.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.886+0000 7fcd3d272640 1 -- 172.21.15.130:0/1627456888 learned_addr learned my addr 172.21.15.130:0/1627456888 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter ms_handle_connect 0x7fcd380b1730 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter ms_handle_connect 0x7fcd3814be90 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter ms_handle_connect 0x7fcd380b4d60 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.887+0000 7fcd2dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1864773568 0 0) 0x7fcd381d7620 con 0x7fcd3814be90 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): my global_id is 4220 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1c004eb0 con 0x7fcd3814be90 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551750930 0 0) 0x7fcd381d7b00 con 0x7fcd380b1730 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd381d7620 con 0x7fcd380b1730 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325794878 0 0) 0x7fcd38142830 con 0x7fcd380b4d60 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T19:42:12.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.889+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd381d7b00 con 0x7fcd380b4d60 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 913387319 0 0) 0x7fcd1c004eb0 con 0x7fcd3814be90 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd38142830 con 0x7fcd3814be90 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcd30001f10 con 0x7fcd3814be90 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:12.894 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3711574775 0 0) 0x7fcd381d7620 con 0x7fcd380b1730 2024-03-25T19:42:12.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd1c004eb0 con 0x7fcd380b1730 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcd340019f0 con 0x7fcd380b1730 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.894+0000 7fcd2dffb640 10 monclient(hunting): dump: 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:12.896 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:12.897 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:12.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.896+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2176838861 0 0) 0x7fcd381d7b00 con 0x7fcd380b4d60 2024-03-25T19:42:12.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.896+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcd381d7620 con 0x7fcd380b4d60 2024-03-25T19:42:12.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2160426330 0 0) 0x7fcd38142830 con 0x7fcd3814be90 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 >> v1:172.21.15.163:6789/0 conn(0x7fcd380b4d60 legacy=0x7fcd38137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 >> v1:172.21.15.130:6789/0 conn(0x7fcd380b1730 legacy=0x7fcd38140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 1 monclient: found mon.c 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd381d77b0 con 0x7fcd3814be90 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 5 monclient: authenticate success, global_id 4220 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 10 monclient: _renew_subs 2024-03-25T19:42:12.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd38142ba0 con 0x7fcd3814be90 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 10 monclient: _renew_subs 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd381432f0 con 0x7fcd3814be90 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.897+0000 7fcd3f4d6640 1 librados: init done 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.898+0000 7fcd3f4d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.898+0000 7fcd3f4d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.898+0000 7fcd3f4d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:12.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:12.898+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd381d7c90 con 0x7fcd3814be90 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcd30003c20 con 0x7fcd3814be90 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcd30003f30 con 0x7fcd3814be90 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2dffb640 10 monclient: dump: 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:13.265 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:13.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.264+0000 7fcd2f7fe640 4 set_mon_vals no callback set 2024-03-25T19:42:13.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.265+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fcd30019f00 con 0x7fcd3814be90 2024-03-25T19:42:13.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.266+0000 7fcd3d272640 10 monclient: get_auth_request con 0x7fcd1c06e1f0 auth_method 0 2024-03-25T19:42:13.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.266+0000 7fcd2dffb640 10 client.4220.objecter ms_handle_connect 0x7fcd1c06e1f0 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.273+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fcd30053870 con 0x7fcd3814be90 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.273+0000 7fcd2dffb640 10 client.4220.objecter ms_dispatch 0x7fcd38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.273+0000 7fcd2dffb640 3 client.4220.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.273+0000 7fcd2dffb640 3 client.4220.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.273+0000 7fcd2dffb640 20 client.4220.objecter dump_active .. 0 homeless 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.274+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcd3004f1b0 con 0x7fcd3814be90 2024-03-25T19:42:13.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.274+0000 7fcd2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:13.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.274+0000 7fcd2dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:13.494 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.495+0000 7fcd3f4d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.495+0000 7fcd3f4d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.495+0000 7fcd3f4d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:13.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.495+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd381d7940 con 0x7fcd3814be90 2024-03-25T19:42:13.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.543+0000 7fcd2dffb640 1 -- 172.21.15.130:0/1627456888 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fcd300512b0 con 0x7fcd3814be90 2024-03-25T19:42:13.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.543+0000 7fcd2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:13.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.543+0000 7fcd2dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:13.543 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:13.543 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:13.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 10 librados: watch_flush enter 2024-03-25T19:42:13.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 10 librados: watch_flush exit 2024-03-25T19:42:13.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 20 client.4220.objecter shutdown clearing up homeless session... 2024-03-25T19:42:13.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 10 client.4220.objecter successfully canceled tick 2024-03-25T19:42:13.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 >> v1:172.21.15.163:6800/1217380380 conn(0x7fcd1c06e1f0 legacy=0x7fcd1c0706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 10 monclient: shutdown 2024-03-25T19:42:13.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:13.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.546+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 >> v1:172.21.15.130:6790/0 conn(0x7fcd3814be90 legacy=0x7fcd381e2e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:13.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.547+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 shutdown_connections 2024-03-25T19:42:13.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.547+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 >> 172.21.15.130:0/1627456888 conn(0x7fcd380617a0 msgr2=0x7fcd380b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:13.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.547+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 shutdown_connections 2024-03-25T19:42:13.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.547+0000 7fcd3f4d6640 1 -- 172.21.15.130:0/1627456888 wait complete. 2024-03-25T19:42:13.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:13.547+0000 7fcd3f4d6640 1 librados: shutdown 2024-03-25T19:42:13.557 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:13.897 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:00:25.606 35429== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.205 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:00:25.927 35425== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.328 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:00:26.051 35424== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:14.451 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:00:26.162 35426== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.116 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:00:26.850 35292== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.174 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:00:26.913 35288== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.215 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:00:26.950 35291== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:15.530 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:00:27.269 35287== Warning: unimplemented fcntl command: 1038 2024-03-25T19:42:20.363 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:20.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.464+0000 7f9313ee8640 10 monclient: get_monmap_and_config 2024-03-25T19:42:20.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.464+0000 7f9313ee8640 10 monclient: build_initial_monmap 2024-03-25T19:42:20.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.465+0000 7f9313ee8640 10 monclient: monmap: 2024-03-25T19:42:20.464 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:20.464 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:20.465821+0000 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:20.465821+0000 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:20.465 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.465+0000 7f9313ee8640 1 Processor -- start 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.465+0000 7f9313ee8640 1 -- start start 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.465+0000 7f9313ee8640 10 monclient: init 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-a con 0x7f930c0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-c con 0x7f930c0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-b con 0x7f930c14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c0af940 con 0x7f930c0b4d60 2024-03-25T19:42:20.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c0b0a70 con 0x7f930c0b1730 2024-03-25T19:42:20.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c0b0d50 con 0x7f930c14be90 2024-03-25T19:42:20.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:20.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.466+0000 7f9313ee8640 10 monclient(hunting): authenticate will time out at 5016.612416s 2024-03-25T19:42:20.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.467+0000 7f9312485640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f930c14be90 0x7f930c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:45956/0 (socket says 172.21.15.130:45956) 2024-03-25T19:42:20.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.467+0000 7f9312485640 1 -- 172.21.15.130:0/4184923347 learned_addr learned my addr 172.21.15.130:0/4184923347 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:20.469 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292146034 0 0) 0x7f930c0af940 con 0x7f930c0b4d60 2024-03-25T19:42:20.469 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.469 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.469 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): my global_id is 4224 2024-03-25T19:42:20.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f8003520 con 0x7f930c0b4d60 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2768416936 0 0) 0x7f930c0b0d50 con 0x7f930c14be90 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 10 monclient(hunting): my global_id is 4223 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.470+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c0af940 con 0x7f930c14be90 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523378346 0 0) 0x7f930c0b0a70 con 0x7f930c0b1730 2024-03-25T19:42:20.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 10 monclient(hunting): my global_id is 4225 2024-03-25T19:42:20.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.471+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c0b0d50 con 0x7f930c0b1730 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2698796462 0 0) 0x7f92f8003520 con 0x7f930c0b4d60 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 >> v1:172.21.15.130:6790/0 conn(0x7f930c14be90 legacy=0x7f930c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 >> v1:172.21.15.163:6789/0 conn(0x7f930c0b1730 legacy=0x7f930c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: _finish_hunting 0 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 monclient: found mon.noname-a 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0b1030 con 0x7f930c0b4d60 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: _finish_auth 0 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: _check_auth_tickets 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9308001690 con 0x7f930c0b4d60 2024-03-25T19:42:20.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9310c82640 10 monclient: dump: 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9313ee8640 5 monclient: authenticate success, global_id 4224 2024-03-25T19:42:20.477 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.474+0000 7f9313ee8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:20.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93080017f0 con 0x7f930c0b4d60 2024-03-25T19:42:20.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:20.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 1 -- 172.21.15.130:0/4184923347 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9308001c00 con 0x7f930c0b4d60 2024-03-25T19:42:20.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9313ee8640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9313ee8640 4 set_mon_vals no callback set 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.477+0000 7f9310c82640 10 monclient: dump: 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.479 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.478+0000 7f9313ee8640 10 monclient: shutdown 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.478+0000 7f9313ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.478+0000 7f9313ee8640 1 -- 172.21.15.130:0/4184923347 >> v1:172.21.15.130:6789/0 conn(0x7f930c0b4d60 legacy=0x7f930c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.479+0000 7f9313ee8640 1 -- 172.21.15.130:0/4184923347 shutdown_connections 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.479+0000 7f9313ee8640 1 -- 172.21.15.130:0/4184923347 >> 172.21.15.130:0/4184923347 conn(0x7f930c0617a0 msgr2=0x7f930c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.479+0000 7f9313ee8640 1 -- 172.21.15.130:0/4184923347 shutdown_connections 2024-03-25T19:42:20.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.479+0000 7f9313ee8640 1 -- 172.21.15.130:0/4184923347 wait complete. 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 10 monclient: build_initial_monmap 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 10 monclient: monmap: 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:20.483 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 librados: starting msgr at 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 librados: starting objecter 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 Processor -- start 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 -- start start 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 librados: setting wanted keys 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 1 librados: calling monclient init 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.480+0000 7f9313ee8640 10 monclient: init 2024-03-25T19:42:20.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-c con 0x7f930c0b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-a con 0x7f930c0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient(hunting): picked mon.noname-b con 0x7f930c14be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c1368f0 con 0x7f930c0b4d60 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c137750 con 0x7f930c14be90 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c1363d0 con 0x7f930c0b1730 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.481+0000 7f9313ee8640 10 monclient(hunting): authenticate will time out at 5016.627507s 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f9311c84640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f930c0b1730 0x7f930c135b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:45970/0 (socket says 172.21.15.130:45970) 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f9311c84640 1 -- 172.21.15.130:0/1368745976 learned_addr learned my addr 172.21.15.130:0/1368745976 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f93027fc640 10 client.?.objecter ms_handle_connect 0x7f930c0b4d60 2024-03-25T19:42:20.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f93027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f93027fc640 10 client.?.objecter ms_handle_connect 0x7f930c0b1730 2024-03-25T19:42:20.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.482+0000 7f93027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.483+0000 7f93027fc640 10 client.?.objecter ms_handle_connect 0x7f930c14be90 2024-03-25T19:42:20.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.483+0000 7f93027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398810553 0 0) 0x7f930c1363d0 con 0x7f930c0b1730 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 10 monclient(hunting): my global_id is 4226 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.484+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92f0004eb0 con 0x7f930c0b1730 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72711000 0 0) 0x7f930c137750 con 0x7f930c14be90 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 10 monclient(hunting): my global_id is 4228 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.485+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c1363d0 con 0x7f930c14be90 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877167569 0 0) 0x7f930c1368f0 con 0x7f930c0b4d60 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 10 monclient(hunting): _negotiate 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 10 monclient(hunting): my global_id is 4227 2024-03-25T19:42:20.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.486+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f930c137750 con 0x7f930c0b4d60 2024-03-25T19:42:20.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281269719 0 0) 0x7f92f0004eb0 con 0x7f930c0b1730 2024-03-25T19:42:20.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f930c1368f0 con 0x7f930c0b1730 2024-03-25T19:42:20.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9308001b40 con 0x7f930c0b1730 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 10 monclient(hunting): dump: 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.491 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.492 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.492 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.492 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.490+0000 7f93027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:20.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3037599762 0 0) 0x7f930c137750 con 0x7f930c0b4d60 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92f0004eb0 con 0x7f930c0b4d60 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92fc001830 con 0x7f930c0b4d60 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): dump: 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.496 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 329465838 0 0) 0x7f930c1363d0 con 0x7f930c14be90 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f930c137750 con 0x7f930c14be90 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9304001c10 con 0x7f930c14be90 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.492+0000 7f93027fc640 10 monclient(hunting): dump: 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.497 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2181103047 0 0) 0x7f930c1368f0 con 0x7f930c0b1730 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 >> v1:172.21.15.163:6789/0 conn(0x7f930c14be90 legacy=0x7f930c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 >> v1:172.21.15.130:6789/0 conn(0x7f930c0b4d60 legacy=0x7f930c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 10 monclient: _finish_hunting 0 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 1 monclient: found mon.c 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c130470 con 0x7f930c0b1730 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 10 monclient: _finish_auth 0 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 10 monclient: _check_auth_tickets 2024-03-25T19:42:20.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f93027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 5 monclient: authenticate success, global_id 4226 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 10 monclient: _renew_subs 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c130720 con 0x7f930c0b1730 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 10 monclient: _renew_subs 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c1322f0 con 0x7f930c0b1730 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.493+0000 7f9313ee8640 1 librados: init done 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.494+0000 7f9313ee8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.494+0000 7f9313ee8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.494+0000 7f9313ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:20.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.494+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c136210 con 0x7f930c0b1730 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93080038a0 con 0x7f930c0b1730 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9308001250 con 0x7f930c0b1730 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f93027fc640 10 monclient: dump: 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:20.580 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:20.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.580+0000 7f9303fff640 4 set_mon_vals no callback set 2024-03-25T19:42:20.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.581+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f9308003d70 con 0x7f930c0b1730 2024-03-25T19:42:20.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.582+0000 7f9311483640 10 monclient: get_auth_request con 0x7f92f006e120 auth_method 0 2024-03-25T19:42:20.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.582+0000 7f93027fc640 10 client.4226.objecter ms_handle_connect 0x7f92f006e120 2024-03-25T19:42:20.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f9308054840 con 0x7f930c0b1730 2024-03-25T19:42:20.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 10 client.4226.objecter ms_dispatch 0x7f930c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 3 client.4226.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 3 client.4226.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 20 client.4226.objecter dump_active .. 0 homeless 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9308050350 con 0x7f930c0b1730 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:20.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:20.840+0000 7f93027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:21.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.120+0000 7f9313ee8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:21.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.120+0000 7f9313ee8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:21.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.120+0000 7f9313ee8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:21.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.120+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f930c136560 con 0x7f930c0b1730 2024-03-25T19:42:21.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.136+0000 7f93027fc640 1 -- 172.21.15.130:0/1368745976 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f9308052280 con 0x7f930c0b1730 2024-03-25T19:42:21.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.136+0000 7f93027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:21.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.136+0000 7f93027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:21.136 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:21.136 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:21.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 10 librados: watch_flush enter 2024-03-25T19:42:21.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 10 librados: watch_flush exit 2024-03-25T19:42:21.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 20 client.4226.objecter shutdown clearing up homeless session... 2024-03-25T19:42:21.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 10 client.4226.objecter successfully canceled tick 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 >> v1:172.21.15.163:6800/1217380380 conn(0x7f92f006e120 legacy=0x7f92f00705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 10 monclient: shutdown 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 >> v1:172.21.15.130:6790/0 conn(0x7f930c0b1730 legacy=0x7f930c135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 shutdown_connections 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 >> 172.21.15.130:0/1368745976 conn(0x7f930c0617a0 msgr2=0x7f930c138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 shutdown_connections 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 -- 172.21.15.130:0/1368745976 wait complete. 2024-03-25T19:42:21.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:21.139+0000 7f9313ee8640 1 librados: shutdown 2024-03-25T19:42:21.152 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:22.608 INFO:tasks.ceph.osd.5.smithi163.stderr:2024-03-25T19:42:22.605+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T19:42:22.780 INFO:tasks.ceph.osd.4.smithi163.stderr:2024-03-25T19:42:22.778+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T19:42:22.845 INFO:tasks.ceph.osd.6.smithi163.stderr:2024-03-25T19:42:22.837+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T19:42:22.897 INFO:tasks.ceph.osd.7.smithi163.stderr:2024-03-25T19:42:22.895+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T19:42:23.941 INFO:tasks.ceph.osd.3.smithi130.stderr:2024-03-25T19:42:23.938+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T19:42:24.005 INFO:tasks.ceph.osd.1.smithi130.stderr:2024-03-25T19:42:24.002+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T19:42:24.105 INFO:tasks.ceph.osd.2.smithi130.stderr:2024-03-25T19:42:24.074+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T19:42:24.785 INFO:tasks.ceph.osd.0.smithi130.stderr:2024-03-25T19:42:24.782+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T19:42:27.959 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:28.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.073+0000 7fcc6efed640 10 monclient: get_monmap_and_config 2024-03-25T19:42:28.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.073+0000 7fcc6efed640 10 monclient: build_initial_monmap 2024-03-25T19:42:28.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.073+0000 7fcc6efed640 10 monclient: monmap: 2024-03-25T19:42:28.073 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:28.073 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:28.074893+0000 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:28.074893+0000 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.074+0000 7fcc6efed640 1 Processor -- start 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.074+0000 7fcc6efed640 1 -- start start 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.074+0000 7fcc6efed640 10 monclient: init 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-c con 0x7fcc680a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-b con 0x7fcc6813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:28.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-a con 0x7fcc681430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:28.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc68142520 con 0x7fcc681430c0 2024-03-25T19:42:28.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680b28a0 con 0x7fcc680a98f0 2024-03-25T19:42:28.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc680b2a30 con 0x7fcc6813f710 2024-03-25T19:42:28.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:28.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.075+0000 7fcc6efed640 10 monclient(hunting): authenticate will time out at 5024.221940s 2024-03-25T19:42:28.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.082+0000 7fcc6cd89640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcc680a98f0 0x7fcc680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:44910/0 (socket says 172.21.15.130:44910) 2024-03-25T19:42:28.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.082+0000 7fcc6cd89640 1 -- 172.21.15.130:0/3186988719 learned_addr learned my addr 172.21.15.130:0/3186988719 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:28.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085897935 0 0) 0x7fcc680b28a0 con 0x7fcc680a98f0 2024-03-25T19:42:28.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 10 monclient(hunting): my global_id is 4255 2024-03-25T19:42:28.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.091+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc58003520 con 0x7fcc680a98f0 2024-03-25T19:42:28.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478060 0 0) 0x7fcc680b2a30 con 0x7fcc6813f710 2024-03-25T19:42:28.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 10 monclient(hunting): my global_id is 4253 2024-03-25T19:42:28.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.101+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc680b28a0 con 0x7fcc6813f710 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34532100 0 0) 0x7fcc58003520 con 0x7fcc680a98f0 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 >> v1:172.21.15.130:6790/0 conn(0x7fcc6813f710 legacy=0x7fcc681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 >> v1:172.21.15.130:6789/0 conn(0x7fcc681430c0 legacy=0x7fcc6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 monclient: found mon.noname-c 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:28.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc680b44c0 con 0x7fcc680a98f0 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc5c001690 con 0x7fcc680a98f0 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc677fe640 10 monclient: dump: 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.111 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.112 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.112 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.112 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc6efed640 5 monclient: authenticate success, global_id 4255 2024-03-25T19:42:28.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.109+0000 7fcc6efed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc5c0017f0 con 0x7fcc680a98f0 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 1 -- 172.21.15.130:0/3186988719 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc5c001c00 con 0x7fcc680a98f0 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc677fe640 10 monclient: dump: 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.128 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.129 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.129 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc6efed640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:28.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.127+0000 7fcc6efed640 4 set_mon_vals no callback set 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.128+0000 7fcc6efed640 10 monclient: shutdown 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.128+0000 7fcc6efed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.128+0000 7fcc6efed640 1 -- 172.21.15.130:0/3186988719 >> v1:172.21.15.163:6789/0 conn(0x7fcc680a98f0 legacy=0x7fcc680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.128+0000 7fcc6efed640 1 -- 172.21.15.130:0/3186988719 shutdown_connections 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.128+0000 7fcc6efed640 1 -- 172.21.15.130:0/3186988719 >> 172.21.15.130:0/3186988719 conn(0x7fcc680b7ea0 msgr2=0x7fcc680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.129+0000 7fcc6efed640 1 -- 172.21.15.130:0/3186988719 shutdown_connections 2024-03-25T19:42:28.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.129+0000 7fcc6efed640 1 -- 172.21.15.130:0/3186988719 wait complete. 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.130+0000 7fcc6efed640 10 monclient: build_initial_monmap 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.130+0000 7fcc6efed640 10 monclient: monmap: 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:28.131 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.130+0000 7fcc6efed640 1 librados: starting msgr at 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.130+0000 7fcc6efed640 1 librados: starting objecter 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.130+0000 7fcc6efed640 1 Processor -- start 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 1 -- start start 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 1 librados: setting wanted keys 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 1 librados: calling monclient init 2024-03-25T19:42:28.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient: init 2024-03-25T19:42:28.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:28.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-a con 0x7fcc680a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-c con 0x7fcc6813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.131+0000 7fcc6efed640 10 monclient(hunting): picked mon.noname-b con 0x7fcc681430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.132+0000 7fcc6efed640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6813b0f0 con 0x7fcc680a98f0 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.132+0000 7fcc6efed640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6813b8f0 con 0x7fcc681430c0 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.132+0000 7fcc6efed640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc6813bb60 con 0x7fcc6813f710 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.132+0000 7fcc6efed640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:28.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.132+0000 7fcc6efed640 10 monclient(hunting): authenticate will time out at 5024.278752s 2024-03-25T19:42:28.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.135+0000 7fcc6cd89640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fcc680a98f0 0x7fcc68139f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:44152/0 (socket says 172.21.15.130:44152) 2024-03-25T19:42:28.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.135+0000 7fcc6cd89640 1 -- 172.21.15.130:0/3784394603 learned_addr learned my addr 172.21.15.130:0/3784394603 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:28.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.136+0000 7fcc657fa640 10 client.?.objecter ms_handle_connect 0x7fcc680a98f0 2024-03-25T19:42:28.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.136+0000 7fcc657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:28.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.139+0000 7fcc657fa640 10 client.?.objecter ms_handle_connect 0x7fcc6813f710 2024-03-25T19:42:28.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.139+0000 7fcc657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:28.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.143+0000 7fcc657fa640 10 client.?.objecter ms_handle_connect 0x7fcc681430c0 2024-03-25T19:42:28.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.143+0000 7fcc657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:28.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743492068 0 0) 0x7fcc6813b8f0 con 0x7fcc681430c0 2024-03-25T19:42:28.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 10 monclient(hunting): my global_id is 4258 2024-03-25T19:42:28.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.148+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc4c004eb0 con 0x7fcc681430c0 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1641238637 0 0) 0x7fcc6813bb60 con 0x7fcc6813f710 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 10 monclient(hunting): my global_id is 4256 2024-03-25T19:42:28.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.150+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6813b8f0 con 0x7fcc6813f710 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665835850 0 0) 0x7fcc6813b0f0 con 0x7fcc680a98f0 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 10 monclient(hunting): _negotiate 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 10 monclient(hunting): my global_id is 4257 2024-03-25T19:42:28.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.152+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc6813bb60 con 0x7fcc680a98f0 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577937977 0 0) 0x7fcc6813b8f0 con 0x7fcc6813f710 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc6813b0f0 con 0x7fcc6813f710 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc54001930 con 0x7fcc6813f710 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 10 monclient(hunting): dump: 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.172 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.173 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.171+0000 7fcc657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 426664764 0 0) 0x7fcc6813bb60 con 0x7fcc680a98f0 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc6813b8f0 con 0x7fcc680a98f0 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc5c001b00 con 0x7fcc680a98f0 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.173+0000 7fcc657fa640 10 monclient(hunting): dump: 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.174 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516936161 0 0) 0x7fcc4c004eb0 con 0x7fcc681430c0 2024-03-25T19:42:28.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc6813bb60 con 0x7fcc681430c0 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc60001b90 con 0x7fcc681430c0 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.175+0000 7fcc657fa640 10 monclient(hunting): dump: 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.176 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1658277333 0 0) 0x7fcc6813b0f0 con 0x7fcc6813f710 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 >> v1:172.21.15.163:6789/0 conn(0x7fcc681430c0 legacy=0x7fcc681e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 >> v1:172.21.15.130:6789/0 conn(0x7fcc680a98f0 legacy=0x7fcc68139f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 10 monclient: _finish_hunting 0 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 1 monclient: found mon.c 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc6813be40 con 0x7fcc6813f710 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 10 monclient: _finish_auth 0 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 10 monclient: _check_auth_tickets 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc6efed640 5 monclient: authenticate success, global_id 4256 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc6efed640 10 monclient: _renew_subs 2024-03-25T19:42:28.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc6efed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.181+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc6814e670 con 0x7fcc6813f710 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.182+0000 7fcc6efed640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.182+0000 7fcc6efed640 10 monclient: _renew_subs 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.182+0000 7fcc6efed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.182+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc6813ca00 con 0x7fcc6813f710 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.182+0000 7fcc6efed640 1 librados: init done 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.183+0000 7fcc6efed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:28.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.183+0000 7fcc6efed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:28.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.183+0000 7fcc6efed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:28.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.183+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc680627b0 con 0x7fcc6813f710 2024-03-25T19:42:28.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc54003610 con 0x7fcc6813f710 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcc54003920 con 0x7fcc6813f710 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc657fa640 10 monclient: dump: 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:28.513 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:28.514 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:28.514 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:28.514 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.512+0000 7fcc66ffd640 4 set_mon_vals no callback set 2024-03-25T19:42:28.514 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.513+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fcc540198f0 con 0x7fcc6813f710 2024-03-25T19:42:28.515 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.513+0000 7fcc6cd89640 10 monclient: get_auth_request con 0x7fcc4c06e0f0 auth_method 0 2024-03-25T19:42:28.515 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.513+0000 7fcc657fa640 10 client.4256.objecter ms_handle_connect 0x7fcc4c06e0f0 2024-03-25T19:42:28.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fcc54053560 con 0x7fcc6813f710 2024-03-25T19:42:28.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 10 client.4256.objecter ms_dispatch 0x7fcc68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:28.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:28.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:28.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T19:42:28.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc5404f070 con 0x7fcc6813f710 2024-03-25T19:42:28.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:28.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.520+0000 7fcc657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:28.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.759+0000 7fcc6efed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.759+0000 7fcc6efed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.759+0000 7fcc6efed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:42:28.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.759+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc681398e0 con 0x7fcc6813f710 2024-03-25T19:42:28.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.849+0000 7fcc657fa640 1 -- 172.21.15.130:0/3784394603 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fcc54050fa0 con 0x7fcc6813f710 2024-03-25T19:42:28.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.850+0000 7fcc657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:28.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.850+0000 7fcc657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:28.851 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:28.851 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 10 librados: watch_flush enter 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 10 librados: watch_flush exit 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 10 client.4256.objecter successfully canceled tick 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 >> v1:172.21.15.163:6800/1217380380 conn(0x7fcc4c06e0f0 legacy=0x7fcc4c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 10 monclient: shutdown 2024-03-25T19:42:28.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 >> v1:172.21.15.130:6790/0 conn(0x7fcc6813f710 legacy=0x7fcc6814bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 shutdown_connections 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 >> 172.21.15.130:0/3784394603 conn(0x7fcc680b7ea0 msgr2=0x7fcc6814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 shutdown_connections 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 -- 172.21.15.130:0/3784394603 wait complete. 2024-03-25T19:42:28.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:28.852+0000 7fcc6efed640 1 librados: shutdown 2024-03-25T19:42:28.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:35.672 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:35.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.794+0000 7f47157fb640 10 monclient: get_monmap_and_config 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.794+0000 7f47157fb640 10 monclient: build_initial_monmap 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.794+0000 7f47157fb640 10 monclient: monmap: 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:35.796278+0000 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:35.796278+0000 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:35.795 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:35.796 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:35.796 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 1 Processor -- start 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 1 -- start start 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient: init 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-a con 0x7f47100a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-b con 0x7f471013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-c con 0x7f47101430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.795+0000 7f47157fb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4710142520 con 0x7f47100a98f0 2024-03-25T19:42:35.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.796+0000 7f47157fb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100b28a0 con 0x7f47101430c0 2024-03-25T19:42:35.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.796+0000 7f47157fb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47100b2a30 con 0x7f471013f710 2024-03-25T19:42:35.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.796+0000 7f47157fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:35.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.796+0000 7f47157fb640 10 monclient(hunting): authenticate will time out at 5031.942684s 2024-03-25T19:42:35.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.808+0000 7f470f7fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f47101430c0 0x7f471014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:49510/0 (socket says 172.21.15.130:49510) 2024-03-25T19:42:35.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.808+0000 7f470f7fe640 1 -- 172.21.15.130:0/2573596389 learned_addr learned my addr 172.21.15.130:0/2573596389 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:35.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 843541784 0 0) 0x7f47100b2a30 con 0x7f471013f710 2024-03-25T19:42:35.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 10 monclient(hunting): my global_id is 4259 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.827+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46f8003520 con 0x7f471013f710 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 514024474 0 0) 0x7f47100b28a0 con 0x7f47101430c0 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 10 monclient(hunting): my global_id is 4261 2024-03-25T19:42:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.828+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47100b2a30 con 0x7f47101430c0 2024-03-25T19:42:35.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 752071403 0 0) 0x7f4710142520 con 0x7f47100a98f0 2024-03-25T19:42:35.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47100b28a0 con 0x7f47100a98f0 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1680882703 0 0) 0x7f46f8003520 con 0x7f471013f710 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 >> v1:172.21.15.163:6789/0 conn(0x7f47101430c0 legacy=0x7f471014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 >> v1:172.21.15.130:6789/0 conn(0x7f47100a98f0 legacy=0x7f47100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 monclient: found mon.noname-b 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47100b44c0 con 0x7f471013f710 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4704001690 con 0x7f471013f710 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f470dffb640 10 monclient: dump: 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:35.839 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:35.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f47157fb640 5 monclient: authenticate success, global_id 4259 2024-03-25T19:42:35.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.836+0000 7f47157fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:35.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47040017f0 con 0x7f471013f710 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 1 -- 172.21.15.130:0/2573596389 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4704001c00 con 0x7f471013f710 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f470dffb640 10 monclient: dump: 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:35.842 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:35.843 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:35.843 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:35.843 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:35.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f47157fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:35.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.841+0000 7f47157fb640 4 set_mon_vals no callback set 2024-03-25T19:42:35.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 10 monclient: shutdown 2024-03-25T19:42:35.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 1 -- 172.21.15.130:0/2573596389 >> v1:172.21.15.130:6790/0 conn(0x7f471013f710 legacy=0x7f47101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 1 -- 172.21.15.130:0/2573596389 shutdown_connections 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 1 -- 172.21.15.130:0/2573596389 >> 172.21.15.130:0/2573596389 conn(0x7f47100b7ea0 msgr2=0x7f47100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 1 -- 172.21.15.130:0/2573596389 shutdown_connections 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.842+0000 7f47157fb640 1 -- 172.21.15.130:0/2573596389 wait complete. 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.843+0000 7f47157fb640 10 monclient: build_initial_monmap 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.843+0000 7f47157fb640 10 monclient: monmap: 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:35.845 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.843+0000 7f47157fb640 1 librados: starting msgr at 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.843+0000 7f47157fb640 1 librados: starting objecter 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.844+0000 7f47157fb640 1 Processor -- start 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.844+0000 7f47157fb640 1 -- start start 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.844+0000 7f47157fb640 1 librados: setting wanted keys 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.844+0000 7f47157fb640 1 librados: calling monclient init 2024-03-25T19:42:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.844+0000 7f47157fb640 10 monclient: init 2024-03-25T19:42:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:42:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-a con 0x7f47100a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-b con 0x7f471013f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient(hunting): picked mon.noname-c con 0x7f47101430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47101d7a10 con 0x7f47100a98f0 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47101d76e0 con 0x7f471013f710 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47101d5210 con 0x7f47101430c0 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.845+0000 7f47157fb640 10 monclient(hunting): authenticate will time out at 5031.991977s 2024-03-25T19:42:35.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.852+0000 7f470e7fc640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f471013f710 0x7f47101d4b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:49550/0 (socket says 172.21.15.130:49550) 2024-03-25T19:42:35.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.852+0000 7f470e7fc640 1 -- 172.21.15.130:0/1380950530 learned_addr learned my addr 172.21.15.130:0/1380950530 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:35.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.853+0000 7f46f77fe640 10 client.?.objecter ms_handle_connect 0x7f471013f710 2024-03-25T19:42:35.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.853+0000 7f46f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 client.?.objecter ms_handle_connect 0x7f47101430c0 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356872453 0 0) 0x7f47101d76e0 con 0x7f471013f710 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 10 monclient(hunting): my global_id is 4264 2024-03-25T19:42:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.860+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc004eb0 con 0x7f471013f710 2024-03-25T19:42:35.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 219020763 0 0) 0x7f46fc004eb0 con 0x7f471013f710 2024-03-25T19:42:35.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47101d76e0 con 0x7f471013f710 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4704001c00 con 0x7f471013f710 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 10 monclient(hunting): dump: 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:35.870 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:35.871 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:35.871 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:35.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.869+0000 7f46f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:35.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851383050 0 0) 0x7f47101d5210 con 0x7f47101430c0 2024-03-25T19:42:35.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:35.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:35.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:35.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 10 monclient(hunting): my global_id is 4262 2024-03-25T19:42:35.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.872+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46fc004eb0 con 0x7f47101430c0 2024-03-25T19:42:35.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.876+0000 7f46f77fe640 10 client.?.objecter ms_handle_connect 0x7f47100a98f0 2024-03-25T19:42:35.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.876+0000 7f46f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2027443588 0 0) 0x7f47101d76e0 con 0x7f471013f710 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 >> v1:172.21.15.130:6790/0 conn(0x7f47101430c0 legacy=0x7f47101384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 >> v1:172.21.15.130:6789/0 conn(0x7f47100a98f0 legacy=0x7f47101d7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 1 monclient: found mon.b 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47101d41e0 con 0x7f471013f710 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.877+0000 7f46f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 5 monclient: authenticate success, global_id 4264 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 10 monclient: _renew_subs 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:35.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47101d4450 con 0x7f471013f710 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 10 monclient: _renew_subs 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4710138dd0 con 0x7f471013f710 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.878+0000 7f47157fb640 1 librados: init done 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.879+0000 7f47157fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.879+0000 7f47157fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:35.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.879+0000 7f47157fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:35.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:35.879+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47101d7870 con 0x7f471013f710 2024-03-25T19:42:36.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4704001230 con 0x7f471013f710 2024-03-25T19:42:36.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4704003960 con 0x7f471013f710 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 10 monclient: dump: 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:36.191 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:36.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f470d7fa640 4 set_mon_vals no callback set 2024-03-25T19:42:36.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.190+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f4704003b80 con 0x7f471013f710 2024-03-25T19:42:36.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.191+0000 7f470effd640 10 monclient: get_auth_request con 0x7f46fc06dd00 auth_method 0 2024-03-25T19:42:36.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.191+0000 7f46f77fe640 10 client.4264.objecter ms_handle_connect 0x7f46fc06dd00 2024-03-25T19:42:36.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.195+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f4704054800 con 0x7f471013f710 2024-03-25T19:42:36.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.195+0000 7f46f77fe640 10 client.4264.objecter ms_dispatch 0x7f4710000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:36.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.195+0000 7f46f77fe640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:36.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.195+0000 7f46f77fe640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:36.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.196+0000 7f46f77fe640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T19:42:36.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.196+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4704050140 con 0x7f471013f710 2024-03-25T19:42:36.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.196+0000 7f46f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:36.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.196+0000 7f46f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:36.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.440+0000 7f47157fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:36.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.440+0000 7f47157fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:36.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.440+0000 7f47157fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:36.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.440+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47101d7ba0 con 0x7f471013f710 2024-03-25T19:42:36.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.463+0000 7f46f77fe640 1 -- 172.21.15.130:0/1380950530 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f4704052240 con 0x7f471013f710 2024-03-25T19:42:36.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.463+0000 7f46f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:36.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.463+0000 7f46f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:36.464 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:36.465 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 10 librados: watch_flush enter 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 10 librados: watch_flush exit 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 10 client.4264.objecter successfully canceled tick 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 >> v1:172.21.15.163:6800/1217380380 conn(0x7f46fc06dd00 legacy=0x7f46fc0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 10 monclient: shutdown 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:36.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 >> v1:172.21.15.163:6789/0 conn(0x7f471013f710 legacy=0x7f47101d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:36.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 shutdown_connections 2024-03-25T19:42:36.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 >> 172.21.15.130:0/1380950530 conn(0x7f47100b7ea0 msgr2=0x7f4710074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:36.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 shutdown_connections 2024-03-25T19:42:36.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 -- 172.21.15.130:0/1380950530 wait complete. 2024-03-25T19:42:36.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:36.466+0000 7f47157fb640 1 librados: shutdown 2024-03-25T19:42:36.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:39.402 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:24.335 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.465 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:24.398 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.497 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:24.398 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.506 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:24.439 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.567 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:24.468 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.571 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:24.505 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.716 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:24.649 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.793 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:24.694 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:39.815 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:24.748 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:43.287 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:43.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.406+0000 7f6db728d640 10 monclient: get_monmap_and_config 2024-03-25T19:42:43.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.406+0000 7f6db728d640 10 monclient: build_initial_monmap 2024-03-25T19:42:43.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.406+0000 7f6db728d640 10 monclient: monmap: 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:43.407539+0000 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:43.407539+0000 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.407+0000 7f6db728d640 1 Processor -- start 2024-03-25T19:42:43.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.407+0000 7f6db728d640 1 -- start start 2024-03-25T19:42:43.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.407+0000 7f6db728d640 10 monclient: init 2024-03-25T19:42:43.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:43.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-b con 0x7f6db00b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-a con 0x7f6db00b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-c con 0x7f6db014be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00af940 con 0x7f6db00b1730 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00b0a70 con 0x7f6db014be90 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db00b0d50 con 0x7f6db00b4d60 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:43.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.408+0000 7f6db728d640 10 monclient(hunting): authenticate will time out at 5039.553805s 2024-03-25T19:42:43.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.416+0000 7f6db5029640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f6db00b4d60 0x7f6db013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:37088/0 (socket says 172.21.15.130:37088) 2024-03-25T19:42:43.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.416+0000 7f6db5029640 1 -- 172.21.15.130:0/4229146474 learned_addr learned my addr 172.21.15.130:0/4229146474 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:43.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714672482 0 0) 0x7f6db00b0a70 con 0x7f6db014be90 2024-03-25T19:42:43.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 10 monclient(hunting): my global_id is 4267 2024-03-25T19:42:43.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.429+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d9c003520 con 0x7f6db014be90 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809320840 0 0) 0x7f6db00b0d50 con 0x7f6db00b4d60 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 10 monclient(hunting): my global_id is 4265 2024-03-25T19:42:43.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.431+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db00b0a70 con 0x7f6db00b4d60 2024-03-25T19:42:43.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191122828 0 0) 0x7f6db00af940 con 0x7f6db00b1730 2024-03-25T19:42:43.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 10 monclient(hunting): my global_id is 4266 2024-03-25T19:42:43.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.435+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db00b0d50 con 0x7f6db00b1730 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 34626264 0 0) 0x7f6d9c003520 con 0x7f6db014be90 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 >> v1:172.21.15.130:6790/0 conn(0x7f6db00b4d60 legacy=0x7f6db013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 >> v1:172.21.15.130:6789/0 conn(0x7f6db00b1730 legacy=0x7f6db00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db00b1030 con 0x7f6db014be90 2024-03-25T19:42:43.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6da8001a10 con 0x7f6db014be90 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6da7fff640 10 monclient: dump: 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.447 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.448 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6db728d640 5 monclient: authenticate success, global_id 4267 2024-03-25T19:42:43.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.446+0000 7f6db728d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:43.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6da8001b70 con 0x7f6db014be90 2024-03-25T19:42:43.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:43.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 1 -- 172.21.15.130:0/4229146474 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6da8001f80 con 0x7f6db014be90 2024-03-25T19:42:43.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:43.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 4 set_mon_vals no callback set 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6da7fff640 10 monclient: dump: 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.450 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 10 monclient: shutdown 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 1 -- 172.21.15.130:0/4229146474 >> v1:172.21.15.163:6789/0 conn(0x7f6db014be90 legacy=0x7f6db014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 1 -- 172.21.15.130:0/4229146474 shutdown_connections 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.450+0000 7f6db728d640 1 -- 172.21.15.130:0/4229146474 >> 172.21.15.130:0/4229146474 conn(0x7f6db00617a0 msgr2=0x7f6db0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.451+0000 7f6db728d640 1 -- 172.21.15.130:0/4229146474 shutdown_connections 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.451+0000 7f6db728d640 1 -- 172.21.15.130:0/4229146474 wait complete. 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 10 monclient: build_initial_monmap 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 10 monclient: monmap: 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:43.453 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 librados: starting msgr at 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 librados: starting objecter 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 Processor -- start 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 -- start start 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 librados: setting wanted keys 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 1 librados: calling monclient init 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.452+0000 7f6db728d640 10 monclient: init 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:43.454 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-c con 0x7f6db00b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-b con 0x7f6db00b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient(hunting): picked mon.noname-a con 0x7f6db014be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db01391b0 con 0x7f6db014be90 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0132750 con 0x7f6db00b4d60 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db0138e40 con 0x7f6db00b1730 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:43.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.453+0000 7f6db728d640 10 monclient(hunting): authenticate will time out at 5039.598903s 2024-03-25T19:42:43.462 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.462+0000 7f6db582a640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f6db014be90 0x7f6db01db2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:35506/0 (socket says 172.21.15.130:35506) 2024-03-25T19:42:43.462 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.462+0000 7f6db582a640 1 -- 172.21.15.130:0/1637840432 learned_addr learned my addr 172.21.15.130:0/1637840432 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:43.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.466+0000 7f6da5ffb640 10 client.?.objecter ms_handle_connect 0x7f6db014be90 2024-03-25T19:42:43.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.466+0000 7f6da5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:43.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.466+0000 7f6da5ffb640 10 client.?.objecter ms_handle_connect 0x7f6db00b1730 2024-03-25T19:42:43.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.466+0000 7f6da5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:43.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.467+0000 7f6da5ffb640 10 client.?.objecter ms_handle_connect 0x7f6db00b4d60 2024-03-25T19:42:43.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.467+0000 7f6da5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 703147935 0 0) 0x7f6db0132750 con 0x7f6db00b4d60 2024-03-25T19:42:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 10 monclient(hunting): my global_id is 4270 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.476+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d94004eb0 con 0x7f6db00b4d60 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224565880 0 0) 0x7f6db0138e40 con 0x7f6db00b1730 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 10 monclient(hunting): my global_id is 4268 2024-03-25T19:42:43.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.477+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0132750 con 0x7f6db00b1730 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998822016 0 0) 0x7f6db01391b0 con 0x7f6db014be90 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 10 monclient(hunting): my global_id is 4269 2024-03-25T19:42:43.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.484+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db0138e40 con 0x7f6db014be90 2024-03-25T19:42:43.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3306933917 0 0) 0x7f6db0132750 con 0x7f6db00b1730 2024-03-25T19:42:43.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db01391b0 con 0x7f6db00b1730 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6dac001970 con 0x7f6db00b1730 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.490 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.490+0000 7f6da5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 482843590 0 0) 0x7f6d94004eb0 con 0x7f6db00b4d60 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6db0132750 con 0x7f6db00b4d60 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6da00018f0 con 0x7f6db00b4d60 2024-03-25T19:42:43.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.491+0000 7f6da5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.492 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3954890260 0 0) 0x7f6db0138e40 con 0x7f6db014be90 2024-03-25T19:42:43.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d94004eb0 con 0x7f6db014be90 2024-03-25T19:42:43.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6da8001e80 con 0x7f6db014be90 2024-03-25T19:42:43.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.499+0000 7f6da5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.499 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3561997279 0 0) 0x7f6db0132750 con 0x7f6db00b4d60 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 >> v1:172.21.15.130:6790/0 conn(0x7f6db00b1730 legacy=0x7f6db0132040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 >> v1:172.21.15.130:6789/0 conn(0x7f6db014be90 legacy=0x7f6db01db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 1 monclient: found mon.b 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:43.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0137c00 con 0x7f6db00b4d60 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6da5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 5 monclient: authenticate success, global_id 4270 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 10 monclient: _renew_subs 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db0138050 con 0x7f6db00b4d60 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 10 monclient: _renew_subs 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db0145080 con 0x7f6db00b4d60 2024-03-25T19:42:43.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.510+0000 7f6db728d640 1 librados: init done 2024-03-25T19:42:43.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.511+0000 7f6db728d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.511+0000 7f6db728d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.511+0000 7f6db728d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:43.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.511+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db01401f0 con 0x7f6db00b4d60 2024-03-25T19:42:43.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.519+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6da0003610 con 0x7f6db00b4d60 2024-03-25T19:42:43.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.519+0000 7f6da5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:43.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.519+0000 7f6da77fe640 4 set_mon_vals no callback set 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.560+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6da0003920 con 0x7f6db00b4d60 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.560+0000 7f6da5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.560+0000 7f6da5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.560+0000 7f6da5ffb640 10 monclient: dump: 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:43.559 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:43.560 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:43.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.808+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f6da00198f0 con 0x7f6db00b4d60 2024-03-25T19:42:43.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.809+0000 7f6db5029640 10 monclient: get_auth_request con 0x7f6d9406e160 auth_method 0 2024-03-25T19:42:43.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.809+0000 7f6da5ffb640 10 client.4270.objecter ms_handle_connect 0x7f6d9406e160 2024-03-25T19:42:43.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.811+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f6da0053680 con 0x7f6db00b4d60 2024-03-25T19:42:43.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.811+0000 7f6da5ffb640 10 client.4270.objecter ms_dispatch 0x7f6db0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:43.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.811+0000 7f6da5ffb640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:43.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.811+0000 7f6da5ffb640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:43.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.811+0000 7f6da5ffb640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T19:42:43.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.813+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6da004efc0 con 0x7f6db00b4d60 2024-03-25T19:42:43.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.814+0000 7f6da5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:43.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:43.814+0000 7f6da5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:44.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.059+0000 7f6db728d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.059+0000 7f6db728d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.059+0000 7f6db728d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:44.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.059+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6db0139360 con 0x7f6db00b4d60 2024-03-25T19:42:44.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.095+0000 7f6da5ffb640 1 -- 172.21.15.130:0/1637840432 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f6db0139360 con 0x7f6db00b4d60 2024-03-25T19:42:44.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.095+0000 7f6da5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:44.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.095+0000 7f6da5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:44.098 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:44.098 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.098+0000 7f6db728d640 10 librados: watch_flush enter 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.098+0000 7f6db728d640 10 librados: watch_flush exit 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.098+0000 7f6db728d640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.098+0000 7f6db728d640 10 client.4270.objecter successfully canceled tick 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 >> v1:172.21.15.163:6800/1217380380 conn(0x7f6d9406e160 legacy=0x7f6d94070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 10 monclient: shutdown 2024-03-25T19:42:44.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 >> v1:172.21.15.163:6789/0 conn(0x7f6db00b4d60 legacy=0x7f6db01dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 shutdown_connections 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 >> 172.21.15.130:0/1637840432 conn(0x7f6db00617a0 msgr2=0x7f6db00b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 shutdown_connections 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 -- 172.21.15.130:0/1637840432 wait complete. 2024-03-25T19:42:44.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:44.099+0000 7f6db728d640 1 librados: shutdown 2024-03-25T19:42:44.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:50.915 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:51.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.032+0000 7f69b7674640 10 monclient: get_monmap_and_config 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.032+0000 7f69b7674640 10 monclient: build_initial_monmap 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.032+0000 7f69b7674640 10 monclient: monmap: 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:51.033201+0000 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:51.033201+0000 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:51.032 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.033+0000 7f69b7674640 1 Processor -- start 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.033+0000 7f69b7674640 1 -- start start 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.033+0000 7f69b7674640 10 monclient: init 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-c con 0x7f69b00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-b con 0x7f69b013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:51.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-a con 0x7f69b01430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0142520 con 0x7f69b01430c0 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b00b28a0 con 0x7f69b00a98f0 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b00b2a30 con 0x7f69b013f710 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.034+0000 7f69b7674640 10 monclient(hunting): authenticate will time out at 5047.180034s 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.035+0000 7f69b5410640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f69b00a98f0 0x7f69b00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:52500/0 (socket says 172.21.15.130:52500) 2024-03-25T19:42:51.035 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.035+0000 7f69b5410640 1 -- 172.21.15.130:0/783430956 learned_addr learned my addr 172.21.15.130:0/783430956 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:51.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264736953 0 0) 0x7f69b00b28a0 con 0x7f69b00a98f0 2024-03-25T19:42:51.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 10 monclient(hunting): my global_id is 4273 2024-03-25T19:42:51.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.126+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f699c003520 con 0x7f69b00a98f0 2024-03-25T19:42:51.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864091391 0 0) 0x7f69b0142520 con 0x7f69b01430c0 2024-03-25T19:42:51.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 10 monclient(hunting): my global_id is 4272 2024-03-25T19:42:51.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.130+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b00b28a0 con 0x7f69b01430c0 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805507170 0 0) 0x7f69b00b2a30 con 0x7f69b013f710 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 10 monclient(hunting): my global_id is 4271 2024-03-25T19:42:51.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.137+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b0142520 con 0x7f69b013f710 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 275063836 0 0) 0x7f699c003520 con 0x7f69b00a98f0 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 >> v1:172.21.15.130:6790/0 conn(0x7f69b013f710 legacy=0x7f69b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 >> v1:172.21.15.130:6789/0 conn(0x7f69b01430c0 legacy=0x7f69b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: _finish_hunting 0 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 monclient: found mon.noname-c 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b00b44c0 con 0x7f69b00a98f0 2024-03-25T19:42:51.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: _finish_auth 0 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: _check_auth_tickets 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69a0001690 con 0x7f69b00a98f0 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69a7fff640 10 monclient: dump: 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.147 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.148 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69b7674640 5 monclient: authenticate success, global_id 4273 2024-03-25T19:42:51.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.146+0000 7f69b7674640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:51.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a00017f0 con 0x7f69b00a98f0 2024-03-25T19:42:51.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:51.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 1 -- 172.21.15.130:0/783430956 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69a0001c00 con 0x7f69b00a98f0 2024-03-25T19:42:51.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69b7674640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:51.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69b7674640 4 set_mon_vals no callback set 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.161+0000 7f69a7fff640 10 monclient: dump: 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.162+0000 7f69b7674640 10 monclient: shutdown 2024-03-25T19:42:51.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.162+0000 7f69b7674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:51.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.162+0000 7f69b7674640 1 -- 172.21.15.130:0/783430956 >> v1:172.21.15.163:6789/0 conn(0x7f69b00a98f0 legacy=0x7f69b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.162+0000 7f69b7674640 1 -- 172.21.15.130:0/783430956 shutdown_connections 2024-03-25T19:42:51.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.162+0000 7f69b7674640 1 -- 172.21.15.130:0/783430956 >> 172.21.15.130:0/783430956 conn(0x7f69b00b7ea0 msgr2=0x7f69b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:51.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.163+0000 7f69b7674640 1 -- 172.21.15.130:0/783430956 shutdown_connections 2024-03-25T19:42:51.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.163+0000 7f69b7674640 1 -- 172.21.15.130:0/783430956 wait complete. 2024-03-25T19:42:51.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.163+0000 7f69b7674640 10 monclient: build_initial_monmap 2024-03-25T19:42:51.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.163+0000 7f69b7674640 10 monclient: monmap: 2024-03-25T19:42:51.165 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:51.165 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:51.165 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 librados: starting msgr at 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 librados: starting objecter 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 Processor -- start 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 -- start start 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 librados: setting wanted keys 2024-03-25T19:42:51.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 librados: calling monclient init 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient: init 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-a con 0x7f69b00a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-c con 0x7f69b013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 10 monclient(hunting): picked mon.noname-b con 0x7f69b01430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0135730 con 0x7f69b00a98f0 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.164+0000 7f69b7674640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b01358c0 con 0x7f69b01430c0 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.165+0000 7f69b7674640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b0135a50 con 0x7f69b013f710 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.165+0000 7f69b7674640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:51.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.165+0000 7f69b7674640 10 monclient(hunting): authenticate will time out at 5047.310636s 2024-03-25T19:42:51.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.173+0000 7f69b5c11640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f69b01430c0 0x7f69b01d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:52548/0 (socket says 172.21.15.130:52548) 2024-03-25T19:42:51.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.173+0000 7f69b5c11640 1 -- 172.21.15.130:0/842481387 learned_addr learned my addr 172.21.15.130:0/842481387 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:51.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.176+0000 7f69a5ffb640 10 client.?.objecter ms_handle_connect 0x7f69b01430c0 2024-03-25T19:42:51.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.176+0000 7f69a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:51.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.176+0000 7f69a5ffb640 10 client.?.objecter ms_handle_connect 0x7f69b00a98f0 2024-03-25T19:42:51.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.176+0000 7f69a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:51.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.183+0000 7f69a5ffb640 10 client.?.objecter ms_handle_connect 0x7f69b013f710 2024-03-25T19:42:51.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.183+0000 7f69a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263629094 0 0) 0x7f69b01358c0 con 0x7f69b01430c0 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.189+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6994004eb0 con 0x7f69b01430c0 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2387870788 0 0) 0x7f69b0135730 con 0x7f69b00a98f0 2024-03-25T19:42:51.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 10 monclient(hunting): my global_id is 4275 2024-03-25T19:42:51.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.190+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69b01358c0 con 0x7f69b00a98f0 2024-03-25T19:42:51.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2127334066 0 0) 0x7f6994004eb0 con 0x7f69b01430c0 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b0135730 con 0x7f69b01430c0 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69a8001c90 con 0x7f69b01430c0 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.202 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.202+0000 7f69a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4228156620 0 0) 0x7f69b0135a50 con 0x7f69b013f710 2024-03-25T19:42:51.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:51.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:51.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:51.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 10 monclient(hunting): my global_id is 4274 2024-03-25T19:42:51.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.204+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6994004eb0 con 0x7f69b013f710 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1327913492 0 0) 0x7f69b01358c0 con 0x7f69b00a98f0 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b0135a50 con 0x7f69b00a98f0 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69a0001b00 con 0x7f69b00a98f0 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.224+0000 7f69a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.224 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.225 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.225 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.225 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.225 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.225 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2260247732 0 0) 0x7f6994004eb0 con 0x7f69b013f710 2024-03-25T19:42:51.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69b01358c0 con 0x7f69b013f710 2024-03-25T19:42:51.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69ac001830 con 0x7f69b013f710 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.229+0000 7f69a5ffb640 10 monclient(hunting): dump: 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.229 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2452021539 0 0) 0x7f69b0135a50 con 0x7f69b00a98f0 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 >> v1:172.21.15.130:6790/0 conn(0x7f69b013f710 legacy=0x7f69b01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 >> v1:172.21.15.163:6789/0 conn(0x7f69b01430c0 legacy=0x7f69b01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 1 monclient: found mon.a 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b01d5000 con 0x7f69b00a98f0 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:51.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 5 monclient: authenticate success, global_id 4275 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 10 monclient: _renew_subs 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b01d52b0 con 0x7f69b00a98f0 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 10 client.4275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 10 monclient: _renew_subs 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b013c900 con 0x7f69b00a98f0 2024-03-25T19:42:51.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.231+0000 7f69b7674640 1 librados: init done 2024-03-25T19:42:51.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.232+0000 7f69b7674640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:51.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.232+0000 7f69b7674640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:51.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.232+0000 7f69b7674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:42:51.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.232+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b013cbb0 con 0x7f69b00a98f0 2024-03-25T19:42:51.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69a0004a90 con 0x7f69b00a98f0 2024-03-25T19:42:51.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f69a0001cc0 con 0x7f69b00a98f0 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a5ffb640 10 monclient: dump: 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:51.527 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:51.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.526+0000 7f69a77fe640 4 set_mon_vals no callback set 2024-03-25T19:42:51.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.527+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f69a001aa70 con 0x7f69b00a98f0 2024-03-25T19:42:51.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.528+0000 7f69b4c0f640 10 monclient: get_auth_request con 0x7f699406e110 auth_method 0 2024-03-25T19:42:51.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.528+0000 7f69a5ffb640 10 client.4275.objecter ms_handle_connect 0x7f699406e110 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f69a0054630 con 0x7f69b00a98f0 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 10 client.4275.objecter ms_dispatch 0x7f69b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 3 client.4275.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 3 client.4275.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 20 client.4275.objecter dump_active .. 0 homeless 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f69a0050140 con 0x7f69b00a98f0 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:51.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.534+0000 7f69a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:51.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.793+0000 7f69b7674640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.793+0000 7f69b7674640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.793+0000 7f69b7674640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:42:51.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.793+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69b01371c0 con 0x7f69b00a98f0 2024-03-25T19:42:51.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.817+0000 7f69a5ffb640 1 -- 172.21.15.130:0/842481387 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f69a0052070 con 0x7f69b00a98f0 2024-03-25T19:42:51.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.817+0000 7f69a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:51.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.817+0000 7f69a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:51.817 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:51.817 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.819+0000 7f69b7674640 10 librados: watch_flush enter 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.819+0000 7f69b7674640 10 librados: watch_flush exit 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.819+0000 7f69b7674640 20 client.4275.objecter shutdown clearing up homeless session... 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.819+0000 7f69b7674640 10 client.4275.objecter successfully canceled tick 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 >> v1:172.21.15.163:6800/1217380380 conn(0x7f699406e110 legacy=0x7f69940705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 10 monclient: shutdown 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 >> v1:172.21.15.130:6789/0 conn(0x7f69b00a98f0 legacy=0x7f69b01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 shutdown_connections 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 >> 172.21.15.130:0/842481387 conn(0x7f69b00b7ea0 msgr2=0x7f69b014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 shutdown_connections 2024-03-25T19:42:51.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 -- 172.21.15.130:0/842481387 wait complete. 2024-03-25T19:42:51.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:51.820+0000 7f69b7674640 1 librados: shutdown 2024-03-25T19:42:51.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:58.639 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.758+0000 7f9fa5425640 10 monclient: get_monmap_and_config 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.758+0000 7f9fa5425640 10 monclient: build_initial_monmap 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.758+0000 7f9fa5425640 10 monclient: monmap: 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:42:58.759236+0000 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:42:58.759236+0000 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:58.758 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:42:58.759 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:42:58.759 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 1 Processor -- start 2024-03-25T19:42:58.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 1 -- start start 2024-03-25T19:42:58.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient: init 2024-03-25T19:42:58.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:58.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:42:58.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:58.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:58.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa01430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:58.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0142520 con 0x7f9fa01430c0 2024-03-25T19:42:58.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00b28a0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa00b2a30 con 0x7f9fa013f710 2024-03-25T19:42:58.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:58.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.759+0000 7f9fa5425640 10 monclient(hunting): authenticate will time out at 5054.905563s 2024-03-25T19:42:58.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.765+0000 7f9f9e7fc640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f9fa013f710 0x7f9fa01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:57196/0 (socket says 172.21.15.130:57196) 2024-03-25T19:42:58.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.765+0000 7f9f9e7fc640 1 -- 172.21.15.130:0/1196160616 learned_addr learned my addr 172.21.15.130:0/1196160616 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 416938759 0 0) 0x7f9fa00b28a0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 10 monclient(hunting): my global_id is 4279 2024-03-25T19:42:58.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.785+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f88003520 con 0x7f9fa00a98f0 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3477402347 0 0) 0x7f9fa00b2a30 con 0x7f9fa013f710 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 10 monclient(hunting): my global_id is 4277 2024-03-25T19:42:58.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.787+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa00b28a0 con 0x7f9fa013f710 2024-03-25T19:42:58.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2377872553 0 0) 0x7f9f88003520 con 0x7f9fa00a98f0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 >> v1:172.21.15.130:6790/0 conn(0x7f9fa013f710 legacy=0x7f9fa01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 >> v1:172.21.15.130:6789/0 conn(0x7f9fa01430c0 legacy=0x7f9fa014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 monclient: found mon.noname-c 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa00b44c0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: _finish_auth 0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f90001690 con 0x7f9fa00a98f0 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9f9dffb640 10 monclient: dump: 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9fa5425640 5 monclient: authenticate success, global_id 4279 2024-03-25T19:42:58.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.798+0000 7f9fa5425640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f900017f0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 1 -- 172.21.15.130:0/1196160616 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f90001c00 con 0x7f9fa00a98f0 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9f9dffb640 10 monclient: dump: 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.803 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9fa5425640 10 monclient: get_monmap_and_config success 2024-03-25T19:42:58.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.803+0000 7f9fa5425640 4 set_mon_vals no callback set 2024-03-25T19:42:58.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 10 monclient: shutdown 2024-03-25T19:42:58.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 1 -- 172.21.15.130:0/1196160616 >> v1:172.21.15.163:6789/0 conn(0x7f9fa00a98f0 legacy=0x7f9fa00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 1 -- 172.21.15.130:0/1196160616 shutdown_connections 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 1 -- 172.21.15.130:0/1196160616 >> 172.21.15.130:0/1196160616 conn(0x7f9fa00b7ea0 msgr2=0x7f9fa00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 1 -- 172.21.15.130:0/1196160616 shutdown_connections 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.804+0000 7f9fa5425640 1 -- 172.21.15.130:0/1196160616 wait complete. 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.805+0000 7f9fa5425640 10 monclient: build_initial_monmap 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.805+0000 7f9fa5425640 10 monclient: monmap: 2024-03-25T19:42:58.808 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.805+0000 7f9fa5425640 1 librados: starting msgr at 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.805+0000 7f9fa5425640 1 librados: starting objecter 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 Processor -- start 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 -- start start 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 librados: setting wanted keys 2024-03-25T19:42:58.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 librados: calling monclient init 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient: init 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient: _reopen_session rank -1 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-b con 0x7f9fa00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-a con 0x7f9fa013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient(hunting): picked mon.noname-c con 0x7f9fa01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0135730 con 0x7f9fa013f710 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa01358c0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fa0135a50 con 0x7f9fa01430c0 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient(hunting): _renew_subs 2024-03-25T19:42:58.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.806+0000 7f9fa5425640 10 monclient(hunting): authenticate will time out at 5054.952506s 2024-03-25T19:42:58.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.812+0000 7f9f9effd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f9fa00a98f0 0x7f9fa01dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:57220/0 (socket says 172.21.15.130:57220) 2024-03-25T19:42:58.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.812+0000 7f9f9effd640 1 -- 172.21.15.130:0/1775582921 learned_addr learned my addr 172.21.15.130:0/1775582921 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:42:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.813+0000 7f9f877fe640 10 client.?.objecter ms_handle_connect 0x7f9fa013f710 2024-03-25T19:42:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.813+0000 7f9f877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.814+0000 7f9f877fe640 10 client.?.objecter ms_handle_connect 0x7f9fa00a98f0 2024-03-25T19:42:58.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.814+0000 7f9f877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.817+0000 7f9f877fe640 10 client.?.objecter ms_handle_connect 0x7f9fa01430c0 2024-03-25T19:42:58.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.817+0000 7f9f877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900327619 0 0) 0x7f9fa0135730 con 0x7f9fa013f710 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 10 monclient(hunting): my global_id is 4281 2024-03-25T19:42:58.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.820+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f8c004eb0 con 0x7f9fa013f710 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.825+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266725551 0 0) 0x7f9fa01358c0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.826+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.826+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.826+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.826+0000 7f9f877fe640 10 monclient(hunting): my global_id is 4282 2024-03-25T19:42:58.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.826+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa0135730 con 0x7f9fa00a98f0 2024-03-25T19:42:58.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603590102 0 0) 0x7f9fa0135a50 con 0x7f9fa01430c0 2024-03-25T19:42:58.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 10 monclient(hunting): _negotiate 2024-03-25T19:42:58.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:42:58.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:42:58.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T19:42:58.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.828+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fa01358c0 con 0x7f9fa01430c0 2024-03-25T19:42:58.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3900792535 0 0) 0x7f9fa01358c0 con 0x7f9fa01430c0 2024-03-25T19:42:58.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa0135a50 con 0x7f9fa01430c0 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f98001a60 con 0x7f9fa01430c0 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 10 monclient(hunting): dump: 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.843+0000 7f9f877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2080571143 0 0) 0x7f9fa0135730 con 0x7f9fa00a98f0 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa01358c0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f90001690 con 0x7f9fa00a98f0 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.844+0000 7f9f877fe640 10 monclient(hunting): dump: 2024-03-25T19:42:58.844 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:58.845 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175091254 0 0) 0x7f9f8c004eb0 con 0x7f9fa013f710 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fa0135730 con 0x7f9fa013f710 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f94001930 con 0x7f9fa013f710 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.846+0000 7f9f877fe640 10 monclient(hunting): dump: 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:58.846 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:58.847 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:58.847 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1667941698 0 0) 0x7f9fa01358c0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 >> v1:172.21.15.130:6790/0 conn(0x7f9fa01430c0 legacy=0x7f9fa01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 >> v1:172.21.15.130:6789/0 conn(0x7f9fa013f710 legacy=0x7f9fa01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 10 monclient: _finish_hunting 0 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 1 monclient: found mon.b 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa01d5000 con 0x7f9fa00a98f0 2024-03-25T19:42:58.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 10 monclient: _finish_auth 0 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 10 monclient: _check_auth_tickets 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9f877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 5 monclient: authenticate success, global_id 4282 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 10 monclient: _renew_subs 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fa01d52b0 con 0x7f9fa00a98f0 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 10 monclient: _renew_subs 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fa013c900 con 0x7f9fa00a98f0 2024-03-25T19:42:58.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.857+0000 7f9fa5425640 1 librados: init done 2024-03-25T19:42:58.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.858+0000 7f9fa5425640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.858+0000 7f9fa5425640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:58.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.858+0000 7f9fa5425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:58.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:58.858+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fa013cbb0 con 0x7f9fa00a98f0 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f900038a0 con 0x7f9fa00a98f0 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f90003a50 con 0x7f9fa00a98f0 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f877fe640 10 monclient: dump: 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:42:59.170 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:42:59.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.170+0000 7f9f9d7fa640 4 set_mon_vals no callback set 2024-03-25T19:42:59.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.171+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f9f90003c70 con 0x7f9fa00a98f0 2024-03-25T19:42:59.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.172+0000 7f9f9e7fc640 10 monclient: get_auth_request con 0x7f9f8c06e160 auth_method 0 2024-03-25T19:42:59.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.172+0000 7f9f877fe640 10 client.4282.objecter ms_handle_connect 0x7f9f8c06e160 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f9f90054730 con 0x7f9fa00a98f0 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 10 client.4282.objecter ms_dispatch 0x7f9fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f90050240 con 0x7f9fa00a98f0 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:42:59.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.176+0000 7f9f877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:42:59.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.433+0000 7f9fa5425640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:59.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.433+0000 7f9fa5425640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:59.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.433+0000 7f9fa5425640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:42:59.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.433+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fa01371c0 con 0x7f9fa00a98f0 2024-03-25T19:42:59.474 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:44.407 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.484 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:44.417 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.533 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:01:44.467 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.545 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:44.479 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.551 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:44.453 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.557 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:44.459 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.558 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:44.491 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.564+0000 7f9f877fe640 1 -- 172.21.15.130:0/1775582921 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f9f90052170 con 0x7f9fa00a98f0 2024-03-25T19:42:59.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.564+0000 7f9f877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:42:59.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.564+0000 7f9f877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:42:59.564 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:42:59.564 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:42:59.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 10 librados: watch_flush enter 2024-03-25T19:42:59.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 10 librados: watch_flush exit 2024-03-25T19:42:59.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T19:42:59.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 10 client.4282.objecter successfully canceled tick 2024-03-25T19:42:59.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 >> v1:172.21.15.163:6800/1217380380 conn(0x7f9f8c06e160 legacy=0x7f9f8c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 10 monclient: shutdown 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 >> v1:172.21.15.163:6789/0 conn(0x7f9fa00a98f0 legacy=0x7f9fa01dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 shutdown_connections 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 >> 172.21.15.130:0/1775582921 conn(0x7f9fa00b7ea0 msgr2=0x7f9fa014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 shutdown_connections 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 -- 172.21.15.130:0/1775582921 wait complete. 2024-03-25T19:42:59.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:42:59.567+0000 7f9fa5425640 1 librados: shutdown 2024-03-25T19:42:59.579 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:42:59.609 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:01:44.511 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:42:59.612 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:01:44.545 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:06.385 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:06.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.503+0000 7f88b4bfb640 10 monclient: get_monmap_and_config 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.503+0000 7f88b4bfb640 10 monclient: build_initial_monmap 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.503+0000 7f88b4bfb640 10 monclient: monmap: 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:06.504126+0000 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:06.504126+0000 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:06.503 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 1 Processor -- start 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 1 -- start start 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient: init 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f88b006c380 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f88b006d670 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f88b00af8f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b006c9a0 con 0x7f88b006c380 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00ada90 con 0x7f88b00af8f0 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b00b0d10 con 0x7f88b006d670 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:06.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.504+0000 7f88b4bfb640 10 monclient(hunting): authenticate will time out at 5062.650413s 2024-03-25T19:43:06.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.509+0000 7f88a7fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f88b006d670 0x7f88b00ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:33512/0 (socket says 172.21.15.130:33512) 2024-03-25T19:43:06.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.509+0000 7f88a7fff640 1 -- 172.21.15.130:0/1101256126 learned_addr learned my addr 172.21.15.130:0/1101256126 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:06.523 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224064583 0 0) 0x7f88b00b0d10 con 0x7f88b006d670 2024-03-25T19:43:06.523 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.523 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.523 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.523 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 10 monclient(hunting): my global_id is 4283 2024-03-25T19:43:06.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.524+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f889c003520 con 0x7f88b006d670 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220131885 0 0) 0x7f88b00ada90 con 0x7f88b00af8f0 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 10 monclient(hunting): my global_id is 4285 2024-03-25T19:43:06.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.530+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b00b0d10 con 0x7f88b00af8f0 2024-03-25T19:43:06.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963240432 0 0) 0x7f88b006c9a0 con 0x7f88b006c380 2024-03-25T19:43:06.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 10 monclient(hunting): my global_id is 4284 2024-03-25T19:43:06.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.537+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b00ada90 con 0x7f88b006c380 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1570666787 0 0) 0x7f889c003520 con 0x7f88b006d670 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 >> v1:172.21.15.163:6789/0 conn(0x7f88b00af8f0 legacy=0x7f88b00aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 >> v1:172.21.15.130:6789/0 conn(0x7f88b006c380 legacy=0x7f88b00b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: _finish_hunting 0 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 monclient: found mon.noname-b 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b00add00 con 0x7f88b006d670 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: _finish_auth 0 2024-03-25T19:43:06.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: _check_auth_tickets 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8898001690 con 0x7f88b006d670 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88add9b640 10 monclient: dump: 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88b4bfb640 5 monclient: authenticate success, global_id 4283 2024-03-25T19:43:06.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.540+0000 7f88b4bfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88980017f0 con 0x7f88b006d670 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 1 -- 172.21.15.130:0/1101256126 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8898001b80 con 0x7f88b006d670 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88b4bfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88b4bfb640 4 set_mon_vals no callback set 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.556+0000 7f88add9b640 10 monclient: dump: 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.556 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 10 monclient: shutdown 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 1 -- 172.21.15.130:0/1101256126 >> v1:172.21.15.130:6790/0 conn(0x7f88b006d670 legacy=0x7f88b00ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.557 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 1 -- 172.21.15.130:0/1101256126 shutdown_connections 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 1 -- 172.21.15.130:0/1101256126 >> 172.21.15.130:0/1101256126 conn(0x7f88b00b5c00 msgr2=0x7f88b00b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 1 -- 172.21.15.130:0/1101256126 shutdown_connections 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.557+0000 7f88b4bfb640 1 -- 172.21.15.130:0/1101256126 wait complete. 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.558+0000 7f88b4bfb640 10 monclient: build_initial_monmap 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.558+0000 7f88b4bfb640 10 monclient: monmap: 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:06.558 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.558+0000 7f88b4bfb640 1 librados: starting msgr at 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.558+0000 7f88b4bfb640 1 librados: starting objecter 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.558+0000 7f88b4bfb640 1 Processor -- start 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 1 -- start start 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 1 librados: setting wanted keys 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 1 librados: calling monclient init 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient: init 2024-03-25T19:43:06.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-c con 0x7f88b006c380 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-b con 0x7f88b006d670 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 10 monclient(hunting): picked mon.noname-a con 0x7f88b00af8f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.559+0000 7f88b4bfb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b01d74c0 con 0x7f88b00af8f0 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.560+0000 7f88b4bfb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b01d8060 con 0x7f88b006d670 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.560+0000 7f88b4bfb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88b0143210 con 0x7f88b006c380 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.560+0000 7f88b4bfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:06.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.560+0000 7f88b4bfb640 10 monclient(hunting): authenticate will time out at 5062.705728s 2024-03-25T19:43:06.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.570+0000 7f88ae59c640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f88b006c380 0x7f88b01d7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:33542/0 (socket says 172.21.15.130:33542) 2024-03-25T19:43:06.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.570+0000 7f88ae59c640 1 -- 172.21.15.130:0/3422714349 learned_addr learned my addr 172.21.15.130:0/3422714349 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:06.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.572+0000 7f88a6ffd640 10 client.?.objecter ms_handle_connect 0x7f88b006c380 2024-03-25T19:43:06.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.572+0000 7f88a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:06.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.572+0000 7f88a6ffd640 10 client.?.objecter ms_handle_connect 0x7f88b006d670 2024-03-25T19:43:06.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.572+0000 7f88a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:06.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.576+0000 7f88a6ffd640 10 client.?.objecter ms_handle_connect 0x7f88b00af8f0 2024-03-25T19:43:06.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.576+0000 7f88a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3149708168 0 0) 0x7f88b01d8060 con 0x7f88b006d670 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): my global_id is 4288 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8890004eb0 con 0x7f88b006d670 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604791687 0 0) 0x7f88b01d74c0 con 0x7f88b00af8f0 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 10 monclient(hunting): my global_id is 4287 2024-03-25T19:43:06.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.581+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b01d8060 con 0x7f88b00af8f0 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804665774 0 0) 0x7f88b0143210 con 0x7f88b006c380 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 10 monclient(hunting): my global_id is 4286 2024-03-25T19:43:06.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.585+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b01d74c0 con 0x7f88b006c380 2024-03-25T19:43:06.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2034169562 0 0) 0x7f88b01d8060 con 0x7f88b00af8f0 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b0143210 con 0x7f88b00af8f0 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f88a8001b50 con 0x7f88b00af8f0 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.592 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.592+0000 7f88a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:06.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281818268 0 0) 0x7f8890004eb0 con 0x7f88b006d670 2024-03-25T19:43:06.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88b01d8060 con 0x7f88b006d670 2024-03-25T19:43:06.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8898001b00 con 0x7f88b006d670 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.601+0000 7f88a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.601 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964530860 0 0) 0x7f88b01d74c0 con 0x7f88b006c380 2024-03-25T19:43:06.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8890004eb0 con 0x7f88b006c380 2024-03-25T19:43:06.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f88a00019b0 con 0x7f88b006c380 2024-03-25T19:43:06.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.614+0000 7f88a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.614 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.620 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2476719240 0 0) 0x7f88b01d8060 con 0x7f88b006d670 2024-03-25T19:43:06.620 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 >> v1:172.21.15.130:6790/0 conn(0x7f88b006c380 legacy=0x7f88b01d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.620 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 >> v1:172.21.15.130:6789/0 conn(0x7f88b00af8f0 legacy=0x7f88b0143b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 1 monclient: found mon.b 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b01433a0 con 0x7f88b006d670 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 5 monclient: authenticate success, global_id 4288 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 10 monclient: _renew_subs 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88b01d4c20 con 0x7f88b006d670 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:06.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 10 monclient: _renew_subs 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88b01d3fe0 con 0x7f88b006d670 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.620+0000 7f88b4bfb640 1 librados: init done 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.622+0000 7f88b4bfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.622+0000 7f88b4bfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:06.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.622+0000 7f88b4bfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:06.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.622+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88b00627b0 con 0x7f88b006d670 2024-03-25T19:43:06.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88980011e0 con 0x7f88b006d670 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8898004c30 con 0x7f88b006d670 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88a6ffd640 10 monclient: dump: 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:06.906 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.906+0000 7f88ad59a640 4 set_mon_vals no callback set 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.907+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f889801ac00 con 0x7f88b006d670 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.907+0000 7f88ae59c640 10 monclient: get_auth_request con 0x7f889006e160 auth_method 0 2024-03-25T19:43:06.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.908+0000 7f88a6ffd640 10 client.4288.objecter ms_handle_connect 0x7f889006e160 2024-03-25T19:43:06.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.911+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f88980547f0 con 0x7f88b006d670 2024-03-25T19:43:06.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.911+0000 7f88a6ffd640 10 client.4288.objecter ms_dispatch 0x7f88b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:06.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.911+0000 7f88a6ffd640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:06.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.911+0000 7f88a6ffd640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:06.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.911+0000 7f88a6ffd640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T19:43:06.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.912+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8898050130 con 0x7f88b006d670 2024-03-25T19:43:06.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.912+0000 7f88a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:06.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:06.912+0000 7f88a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:07.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.168+0000 7f88b4bfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.168+0000 7f88b4bfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.168+0000 7f88b4bfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:07.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.168+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88b01d7670 con 0x7f88b006d670 2024-03-25T19:43:07.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.189+0000 7f88a6ffd640 1 -- 172.21.15.130:0/3422714349 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f88b01d7670 con 0x7f88b006d670 2024-03-25T19:43:07.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.189+0000 7f88a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:07.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.189+0000 7f88a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:07.189 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:07.189 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 10 librados: watch_flush enter 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 10 librados: watch_flush exit 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 10 client.4288.objecter successfully canceled tick 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 >> v1:172.21.15.163:6800/1217380380 conn(0x7f889006e160 legacy=0x7f8890070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 10 monclient: shutdown 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 >> v1:172.21.15.163:6789/0 conn(0x7f88b006d670 legacy=0x7f88b01d4510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:07.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 shutdown_connections 2024-03-25T19:43:07.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 >> 172.21.15.130:0/3422714349 conn(0x7f88b00b5c00 msgr2=0x7f88b0148460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:07.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 shutdown_connections 2024-03-25T19:43:07.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 -- 172.21.15.130:0/3422714349 wait complete. 2024-03-25T19:43:07.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:07.192+0000 7f88b4bfb640 1 librados: shutdown 2024-03-25T19:43:07.202 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:14.009 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:14.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.113+0000 7f92bdd8f640 10 monclient: get_monmap_and_config 2024-03-25T19:43:14.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.113+0000 7f92bdd8f640 10 monclient: build_initial_monmap 2024-03-25T19:43:14.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.113+0000 7f92bdd8f640 10 monclient: monmap: 2024-03-25T19:43:14.112 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:14.114089+0000 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:14.114089+0000 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:14.113 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.113+0000 7f92bdd8f640 1 Processor -- start 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 1 -- start start 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient: init 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:14.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-a con 0x7f92b80a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-b con 0x7f92b813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-c con 0x7f92b81430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b8142520 con 0x7f92b80a98f0 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80b28a0 con 0x7f92b81430c0 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b80b2a30 con 0x7f92b813f710 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.114+0000 7f92bdd8f640 10 monclient(hunting): authenticate will time out at 5070.260196s 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.115+0000 7f92b6ffd640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f92b813f710 0x7f92b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:40986/0 (socket says 172.21.15.130:40986) 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.115+0000 7f92b7fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f92b81430c0 0x7f92b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:37566/0 (socket says 172.21.15.130:37566) 2024-03-25T19:43:14.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.115+0000 7f92b6ffd640 1 -- 172.21.15.130:0/1566476205 learned_addr learned my addr 172.21.15.130:0/1566476205 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629657942 0 0) 0x7f92b8142520 con 0x7f92b80a98f0 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 10 monclient(hunting): my global_id is 4290 2024-03-25T19:43:14.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.156+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a0003520 con 0x7f92b80a98f0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2928911895 0 0) 0x7f92a0003520 con 0x7f92b80a98f0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 >> v1:172.21.15.130:6790/0 conn(0x7f92b813f710 legacy=0x7f92b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 >> v1:172.21.15.163:6789/0 conn(0x7f92b81430c0 legacy=0x7f92b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 monclient: found mon.noname-a 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b80b44c0 con 0x7f92b80a98f0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: _finish_auth 0 2024-03-25T19:43:14.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92a8001690 con 0x7f92b80a98f0 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.172+0000 7f92b67fc640 10 monclient: dump: 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.173+0000 7f92bdd8f640 5 monclient: authenticate success, global_id 4290 2024-03-25T19:43:14.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.173+0000 7f92bdd8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:14.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92a80017f0 con 0x7f92b80a98f0 2024-03-25T19:43:14.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:14.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 1 -- 172.21.15.130:0/1566476205 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92a8001c00 con 0x7f92b80a98f0 2024-03-25T19:43:14.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92b67fc640 10 monclient: dump: 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92bdd8f640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:14.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92bdd8f640 4 set_mon_vals no callback set 2024-03-25T19:43:14.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92bdd8f640 10 monclient: shutdown 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92bdd8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.204+0000 7f92bdd8f640 1 -- 172.21.15.130:0/1566476205 >> v1:172.21.15.130:6789/0 conn(0x7f92b80a98f0 legacy=0x7f92b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 -- 172.21.15.130:0/1566476205 shutdown_connections 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 -- 172.21.15.130:0/1566476205 >> 172.21.15.130:0/1566476205 conn(0x7f92b80b7ea0 msgr2=0x7f92b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 -- 172.21.15.130:0/1566476205 shutdown_connections 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 -- 172.21.15.130:0/1566476205 wait complete. 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 10 monclient: build_initial_monmap 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 10 monclient: monmap: 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:14.206 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 librados: starting msgr at 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.205+0000 7f92bdd8f640 1 librados: starting objecter 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.206+0000 7f92bdd8f640 1 Processor -- start 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.206+0000 7f92bdd8f640 1 -- start start 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.206+0000 7f92bdd8f640 1 librados: setting wanted keys 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.206+0000 7f92bdd8f640 1 librados: calling monclient init 2024-03-25T19:43:14.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.206+0000 7f92bdd8f640 10 monclient: init 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-a con 0x7f92b80a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-b con 0x7f92b813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient(hunting): picked mon.noname-c con 0x7f92b81430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b8135ac0 con 0x7f92b80a98f0 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b8137030 con 0x7f92b813f710 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92b81372a0 con 0x7f92b81430c0 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:14.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.207+0000 7f92bdd8f640 10 monclient(hunting): authenticate will time out at 5070.352741s 2024-03-25T19:43:14.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.209+0000 7f92b6ffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f92b813f710 0x7f92b81379a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:37592/0 (socket says 172.21.15.130:37592) 2024-03-25T19:43:14.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.209+0000 7f92b6ffd640 1 -- 172.21.15.130:0/2744955218 learned_addr learned my addr 172.21.15.130:0/2744955218 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:14.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.210+0000 7f929ffff640 10 client.?.objecter ms_handle_connect 0x7f92b813f710 2024-03-25T19:43:14.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.210+0000 7f929ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.216+0000 7f929ffff640 10 client.?.objecter ms_handle_connect 0x7f92b81430c0 2024-03-25T19:43:14.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.216+0000 7f929ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457534610 0 0) 0x7f92b8137030 con 0x7f92b813f710 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 10 monclient(hunting): my global_id is 4294 2024-03-25T19:43:14.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.218+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a4004eb0 con 0x7f92b813f710 2024-03-25T19:43:14.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.225+0000 7f929ffff640 10 client.?.objecter ms_handle_connect 0x7f92b80a98f0 2024-03-25T19:43:14.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.225+0000 7f929ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:14.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2126467506 0 0) 0x7f92a4004eb0 con 0x7f92b813f710 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b8137030 con 0x7f92b813f710 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92ac0018b0 con 0x7f92b813f710 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:14.233 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.233+0000 7f929ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:14.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821513552 0 0) 0x7f92b8135ac0 con 0x7f92b80a98f0 2024-03-25T19:43:14.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 10 monclient(hunting): my global_id is 4293 2024-03-25T19:43:14.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.237+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92a4004eb0 con 0x7f92b80a98f0 2024-03-25T19:43:14.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150987865 0 0) 0x7f92b81372a0 con 0x7f92b81430c0 2024-03-25T19:43:14.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:14.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:14.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:14.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 10 monclient(hunting): my global_id is 4292 2024-03-25T19:43:14.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.240+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f92b8135ac0 con 0x7f92b81430c0 2024-03-25T19:43:14.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1499530141 0 0) 0x7f92a4004eb0 con 0x7f92b80a98f0 2024-03-25T19:43:14.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f92b81372a0 con 0x7f92b80a98f0 2024-03-25T19:43:14.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92a8001bc0 con 0x7f92b80a98f0 2024-03-25T19:43:14.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.247+0000 7f929ffff640 10 monclient(hunting): dump: 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:14.247 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1669218609 0 0) 0x7f92b8137030 con 0x7f92b813f710 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 >> v1:172.21.15.130:6790/0 conn(0x7f92b81430c0 legacy=0x7f92b81e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 >> v1:172.21.15.130:6789/0 conn(0x7f92b80a98f0 legacy=0x7f92b8133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 1 monclient: found mon.b 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8137580 con 0x7f92b813f710 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 10 monclient: _finish_auth 0 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f929ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 5 monclient: authenticate success, global_id 4294 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 10 monclient: _renew_subs 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:14.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92b81363b0 con 0x7f92b813f710 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 10 monclient: _renew_subs 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f92b81353a0 con 0x7f92b813f710 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.254+0000 7f92bdd8f640 1 librados: init done 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.255+0000 7f92bdd8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:14.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.255+0000 7f92bdd8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:14.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.255+0000 7f92bdd8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:14.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.255+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92b8130d90 con 0x7f92b813f710 2024-03-25T19:43:14.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f92ac0047e0 con 0x7f92b813f710 2024-03-25T19:43:14.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f92ac004a90 con 0x7f92b813f710 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f929ffff640 10 monclient: dump: 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:14.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.275+0000 7f92b5ffb640 4 set_mon_vals no callback set 2024-03-25T19:43:14.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.276+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f92ac004cb0 con 0x7f92b813f710 2024-03-25T19:43:14.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.277+0000 7f92b77fe640 10 monclient: get_auth_request con 0x7f92a406e020 auth_method 0 2024-03-25T19:43:14.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.277+0000 7f929ffff640 10 client.4294.objecter ms_handle_connect 0x7f92a406e020 2024-03-25T19:43:14.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.279+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f92ac0546f0 con 0x7f92b813f710 2024-03-25T19:43:14.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.279+0000 7f929ffff640 10 client.4294.objecter ms_dispatch 0x7f92b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:14.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.279+0000 7f929ffff640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:14.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.279+0000 7f929ffff640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:14.278 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.279+0000 7f929ffff640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T19:43:14.565 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.566+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f92ac050200 con 0x7f92b813f710 2024-03-25T19:43:14.565 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.566+0000 7f929ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:14.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.566+0000 7f929ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:14.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.815+0000 7f92bdd8f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.815+0000 7f92bdd8f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.815+0000 7f92bdd8f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:14.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.815+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92b8074cc0 con 0x7f92b813f710 2024-03-25T19:43:14.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.845+0000 7f929ffff640 1 -- 172.21.15.130:0/2744955218 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f92ac052130 con 0x7f92b813f710 2024-03-25T19:43:14.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.845+0000 7f929ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:14.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.845+0000 7f929ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:14.845 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:14.845 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 10 librados: watch_flush enter 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 10 librados: watch_flush exit 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 10 client.4294.objecter successfully canceled tick 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 >> v1:172.21.15.163:6800/1217380380 conn(0x7f92a406e020 legacy=0x7f92a40704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 10 monclient: shutdown 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:14.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 >> v1:172.21.15.163:6789/0 conn(0x7f92b813f710 legacy=0x7f92b81379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:14.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 shutdown_connections 2024-03-25T19:43:14.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 >> 172.21.15.130:0/2744955218 conn(0x7f92b80b7ea0 msgr2=0x7f92b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:14.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.848+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 shutdown_connections 2024-03-25T19:43:14.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.849+0000 7f92bdd8f640 1 -- 172.21.15.130:0/2744955218 wait complete. 2024-03-25T19:43:14.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:14.849+0000 7f92bdd8f640 1 librados: shutdown 2024-03-25T19:43:14.861 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:19.581 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:04.514 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.597 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:04.530 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.624 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:04.558 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.632 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:04.533 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.642 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:04.543 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.656 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:04.589 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.663 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:04.564 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.667 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:04.600 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:19.706 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:04.639 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:21.667 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:21.784 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.784+0000 7f81b074d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.784+0000 7f81b074d640 10 monclient: build_initial_monmap 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.784+0000 7f81b074d640 10 monclient: monmap: 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:21.786185+0000 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:21.786185+0000 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:21.785 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.785+0000 7f81b074d640 1 Processor -- start 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.785+0000 7f81b074d640 1 -- start start 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.785+0000 7f81b074d640 10 monclient: init 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-a con 0x7f81a80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-c con 0x7f81a80b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-b con 0x7f81a814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:21.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80af940 con 0x7f81a80b4d60 2024-03-25T19:43:21.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80b0a70 con 0x7f81a80b1730 2024-03-25T19:43:21.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a80b0d50 con 0x7f81a814be90 2024-03-25T19:43:21.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:21.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.786+0000 7f81b074d640 10 monclient(hunting): authenticate will time out at 5077.933103s 2024-03-25T19:43:21.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.795+0000 7f81ae4e9640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f81a80b4d60 0x7f81a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:48106/0 (socket says 172.21.15.130:48106) 2024-03-25T19:43:21.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.795+0000 7f81ae4e9640 1 -- 172.21.15.130:0/2124067214 learned_addr learned my addr 172.21.15.130:0/2124067214 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894244417 0 0) 0x7f81a80af940 con 0x7f81a80b4d60 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 10 monclient(hunting): my global_id is 4296 2024-03-25T19:43:21.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.808+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8194003520 con 0x7f81a80b4d60 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988545819 0 0) 0x7f81a80b0a70 con 0x7f81a80b1730 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 10 monclient(hunting): my global_id is 4297 2024-03-25T19:43:21.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.815+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a80af940 con 0x7f81a80b1730 2024-03-25T19:43:21.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960986160 0 0) 0x7f81a80b0d50 con 0x7f81a814be90 2024-03-25T19:43:21.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 10 monclient(hunting): my global_id is 4295 2024-03-25T19:43:21.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.817+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a80b0a70 con 0x7f81a814be90 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765655747 0 0) 0x7f81a80af940 con 0x7f81a80b1730 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 >> v1:172.21.15.130:6790/0 conn(0x7f81a814be90 legacy=0x7f81a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 >> v1:172.21.15.130:6789/0 conn(0x7f81a80b4d60 legacy=0x7f81a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: _finish_hunting 0 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 monclient: found mon.noname-c 2024-03-25T19:43:21.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a80b1030 con 0x7f81a80b1730 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: _finish_auth 0 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: _check_auth_tickets 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8198001670 con 0x7f81a80b1730 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81ad4e7640 10 monclient: dump: 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.830 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81b074d640 5 monclient: authenticate success, global_id 4297 2024-03-25T19:43:21.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.828+0000 7f81b074d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81980017d0 con 0x7f81a80b1730 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 1 -- 172.21.15.130:0/2124067214 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8198001be0 con 0x7f81a80b1730 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81b074d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81b074d640 4 set_mon_vals no callback set 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.843+0000 7f81ad4e7640 10 monclient: dump: 2024-03-25T19:43:21.844 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:21.845 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.844+0000 7f81b074d640 10 monclient: shutdown 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.844+0000 7f81b074d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.844+0000 7f81b074d640 1 -- 172.21.15.130:0/2124067214 >> v1:172.21.15.163:6789/0 conn(0x7f81a80b1730 legacy=0x7f81a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.844+0000 7f81b074d640 1 -- 172.21.15.130:0/2124067214 shutdown_connections 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.844+0000 7f81b074d640 1 -- 172.21.15.130:0/2124067214 >> 172.21.15.130:0/2124067214 conn(0x7f81a80617a0 msgr2=0x7f81a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.845+0000 7f81b074d640 1 -- 172.21.15.130:0/2124067214 shutdown_connections 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.845+0000 7f81b074d640 1 -- 172.21.15.130:0/2124067214 wait complete. 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 10 monclient: build_initial_monmap 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 10 monclient: monmap: 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:21.847 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 librados: starting msgr at 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 librados: starting objecter 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 Processor -- start 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 -- start start 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 librados: setting wanted keys 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 1 librados: calling monclient init 2024-03-25T19:43:21.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.846+0000 7f81b074d640 10 monclient: init 2024-03-25T19:43:21.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:21.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-b con 0x7f81a80b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-a con 0x7f81a80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient(hunting): picked mon.noname-c con 0x7f81a814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813a4c0 con 0x7f81a80b4d60 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813a190 con 0x7f81a80b1730 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81a813b6e0 con 0x7f81a814be90 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:21.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.847+0000 7f81b074d640 10 monclient(hunting): authenticate will time out at 5077.994040s 2024-03-25T19:43:21.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.854+0000 7f81ae4e9640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f81a80b1730 0x7f81a8139a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:53184/0 (socket says 172.21.15.130:53184) 2024-03-25T19:43:21.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.854+0000 7f81ae4e9640 1 -- 172.21.15.130:0/2763205976 learned_addr learned my addr 172.21.15.130:0/2763205976 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:21.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.856+0000 7f819effd640 10 client.?.objecter ms_handle_connect 0x7f81a80b1730 2024-03-25T19:43:21.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.856+0000 7f819effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.859+0000 7f819effd640 10 client.?.objecter ms_handle_connect 0x7f81a814be90 2024-03-25T19:43:21.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.859+0000 7f819effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114459831 0 0) 0x7f81a813a190 con 0x7f81a80b1730 2024-03-25T19:43:21.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 10 monclient(hunting): my global_id is 4300 2024-03-25T19:43:21.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.861+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a80b1730 2024-03-25T19:43:21.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.864+0000 7f819effd640 10 client.?.objecter ms_handle_connect 0x7f81a80b4d60 2024-03-25T19:43:21.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.864+0000 7f819effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373299079 0 0) 0x7f81a813b6e0 con 0x7f81a814be90 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): my global_id is 4298 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a813a190 con 0x7f81a814be90 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738466717 0 0) 0x7f81a813a4c0 con 0x7f81a80b4d60 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:21.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:21.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 10 monclient(hunting): my global_id is 4299 2024-03-25T19:43:21.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.872+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81a813b6e0 con 0x7f81a80b4d60 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2134436655 0 0) 0x7f818c004eb0 con 0x7f81a80b1730 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81a813a4c0 con 0x7f81a80b1730 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f81a4001950 con 0x7f81a80b1730 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 10 monclient(hunting): dump: 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.875 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.874+0000 7f819effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.885+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 793142814 0 0) 0x7f81a813a190 con 0x7f81a814be90 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.886+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f818c004eb0 con 0x7f81a814be90 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.886+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f81a0001b50 con 0x7f81a814be90 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.886+0000 7f819effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.886+0000 7f819effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.886+0000 7f819effd640 10 monclient(hunting): dump: 2024-03-25T19:43:21.886 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:21.887 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4173984691 0 0) 0x7f81a813a4c0 con 0x7f81a80b1730 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 >> v1:172.21.15.130:6790/0 conn(0x7f81a814be90 legacy=0x7f81a81e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 >> v1:172.21.15.130:6789/0 conn(0x7f81a80b4d60 legacy=0x7f81a81d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 1 monclient: found mon.b 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81a8139010 con 0x7f81a80b1730 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 10 monclient: _finish_auth 0 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f819effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:21.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f81b074d640 5 monclient: authenticate success, global_id 4300 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f81b074d640 10 monclient: _renew_subs 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f81b074d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.889+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81a813bb00 con 0x7f81a80b1730 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 monclient: _renew_subs 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81a81d80f0 con 0x7f81a80b1730 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 1 librados: init done 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:21.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:21.890+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81a813be00 con 0x7f81a80b1730 2024-03-25T19:43:22.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81a40047e0 con 0x7f81a80b1730 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f81acce6640 4 set_mon_vals no callback set 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f81a4004af0 con 0x7f81a80b1730 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.175+0000 7f819effd640 10 monclient: dump: 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:22.176 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:22.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.176+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f81a401aac0 con 0x7f81a80b1730 2024-03-25T19:43:22.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.176+0000 7f81adce8640 10 monclient: get_auth_request con 0x7f818c06df70 auth_method 0 2024-03-25T19:43:22.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.177+0000 7f819effd640 10 client.4300.objecter ms_handle_connect 0x7f818c06df70 2024-03-25T19:43:22.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.183+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f81a40548d0 con 0x7f81a80b1730 2024-03-25T19:43:22.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.183+0000 7f819effd640 10 client.4300.objecter ms_dispatch 0x7f81a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:22.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.183+0000 7f819effd640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:22.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.183+0000 7f819effd640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:22.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.183+0000 7f819effd640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T19:43:22.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.185+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81a4050210 con 0x7f81a80b1730 2024-03-25T19:43:22.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.185+0000 7f819effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:22.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.185+0000 7f819effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:22.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.406+0000 7f81b074d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:22.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.406+0000 7f81b074d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:22.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.406+0000 7f81b074d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:43:22.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.406+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81a80afbd0 con 0x7f81a80b1730 2024-03-25T19:43:22.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.440+0000 7f819effd640 1 -- 172.21.15.130:0/2763205976 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f81a4052310 con 0x7f81a80b1730 2024-03-25T19:43:22.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.441+0000 7f819effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:22.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.441+0000 7f819effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:22.442 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:22.442 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.443+0000 7f81b074d640 10 librados: watch_flush enter 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 10 librados: watch_flush exit 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 10 client.4300.objecter successfully canceled tick 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 >> v1:172.21.15.163:6800/1217380380 conn(0x7f818c06df70 legacy=0x7f818c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 10 monclient: shutdown 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 >> v1:172.21.15.163:6789/0 conn(0x7f81a80b1730 legacy=0x7f81a8139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 shutdown_connections 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 >> 172.21.15.130:0/2763205976 conn(0x7f81a80617a0 msgr2=0x7f81a80b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:22.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 shutdown_connections 2024-03-25T19:43:22.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 -- 172.21.15.130:0/2763205976 wait complete. 2024-03-25T19:43:22.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:22.444+0000 7f81b074d640 1 librados: shutdown 2024-03-25T19:43:22.456 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:29.262 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.375+0000 7f886499d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.375+0000 7f886499d640 10 monclient: build_initial_monmap 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.375+0000 7f886499d640 10 monclient: monmap: 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:29.377573+0000 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:29.377573+0000 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:29.376 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:29.377 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:29.377 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.376+0000 7f886499d640 1 Processor -- start 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.376+0000 7f886499d640 1 -- start start 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.376+0000 7f886499d640 10 monclient: init 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient(hunting): picked mon.noname-c con 0x7f88600b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient(hunting): picked mon.noname-b con 0x7f88600b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient(hunting): picked mon.noname-a con 0x7f886014be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600af940 con 0x7f886014be90 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600b0a70 con 0x7f88600b4d60 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600b0d50 con 0x7f88600b1730 2024-03-25T19:43:29.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:29.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.377+0000 7f886499d640 10 monclient(hunting): authenticate will time out at 5085.523973s 2024-03-25T19:43:29.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.385+0000 7f885e59c640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f88600b4d60 0x7f886013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:44230/0 (socket says 172.21.15.130:44230) 2024-03-25T19:43:29.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.385+0000 7f885e59c640 1 -- 172.21.15.130:0/3787570511 learned_addr learned my addr 172.21.15.130:0/3787570511 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3138459050 0 0) 0x7f88600af940 con 0x7f886014be90 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 10 monclient(hunting): my global_id is 4302 2024-03-25T19:43:29.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.398+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8848003520 con 0x7f886014be90 2024-03-25T19:43:29.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 388973878 0 0) 0x7f88600b0a70 con 0x7f88600b4d60 2024-03-25T19:43:29.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 10 monclient(hunting): my global_id is 4303 2024-03-25T19:43:29.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.402+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88600af940 con 0x7f88600b4d60 2024-03-25T19:43:29.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804253650 0 0) 0x7f88600b0d50 con 0x7f88600b1730 2024-03-25T19:43:29.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 10 monclient(hunting): my global_id is 4301 2024-03-25T19:43:29.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.412+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88600b0a70 con 0x7f88600b1730 2024-03-25T19:43:29.419 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434470532 0 0) 0x7f8848003520 con 0x7f886014be90 2024-03-25T19:43:29.419 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 >> v1:172.21.15.130:6790/0 conn(0x7f88600b1730 legacy=0x7f88600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.419 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 >> v1:172.21.15.163:6789/0 conn(0x7f88600b4d60 legacy=0x7f886013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.419 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 monclient: found mon.noname-a 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600b1030 con 0x7f886014be90 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: _finish_auth 0 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8858001a10 con 0x7f886014be90 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f885d59a640 10 monclient: dump: 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.420 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f886499d640 5 monclient: authenticate success, global_id 4302 2024-03-25T19:43:29.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.418+0000 7f886499d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:29.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8858001b70 con 0x7f886014be90 2024-03-25T19:43:29.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:29.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 1 -- 172.21.15.130:0/3787570511 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8858001f80 con 0x7f886014be90 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f886499d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f886499d640 4 set_mon_vals no callback set 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.436+0000 7f885d59a640 10 monclient: dump: 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.437 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.438 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 10 monclient: shutdown 2024-03-25T19:43:29.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 1 -- 172.21.15.130:0/3787570511 >> v1:172.21.15.130:6789/0 conn(0x7f886014be90 legacy=0x7f886014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 1 -- 172.21.15.130:0/3787570511 shutdown_connections 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 1 -- 172.21.15.130:0/3787570511 >> 172.21.15.130:0/3787570511 conn(0x7f88600617a0 msgr2=0x7f8860061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 1 -- 172.21.15.130:0/3787570511 shutdown_connections 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.437+0000 7f886499d640 1 -- 172.21.15.130:0/3787570511 wait complete. 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 10 monclient: build_initial_monmap 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 10 monclient: monmap: 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:29.440 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 librados: starting msgr at 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 librados: starting objecter 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 Processor -- start 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 -- start start 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 librados: setting wanted keys 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 1 librados: calling monclient init 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.438+0000 7f886499d640 10 monclient: init 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient(hunting): picked mon.noname-b con 0x7f88600b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:29.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient(hunting): picked mon.noname-c con 0x7f88600b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient(hunting): picked mon.noname-a con 0x7f886014be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8860142710 con 0x7f886014be90 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8860142410 con 0x7f88600b1730 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8860140d00 con 0x7f88600b4d60 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:29.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.439+0000 7f886499d640 10 monclient(hunting): authenticate will time out at 5085.586039s 2024-03-25T19:43:29.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.448+0000 7f885dd9b640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f88600b4d60 0x7f8860142ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:36474/0 (socket says 172.21.15.130:36474) 2024-03-25T19:43:29.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.448+0000 7f885dd9b640 1 -- 172.21.15.130:0/2075503886 learned_addr learned my addr 172.21.15.130:0/2075503886 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:29.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.452+0000 7f884effd640 10 client.?.objecter ms_handle_connect 0x7f88600b4d60 2024-03-25T19:43:29.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.452+0000 7f884effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:29.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.455+0000 7f884effd640 10 client.?.objecter ms_handle_connect 0x7f88600b1730 2024-03-25T19:43:29.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.455+0000 7f884effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:29.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.459+0000 7f884effd640 10 client.?.objecter ms_handle_connect 0x7f886014be90 2024-03-25T19:43:29.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.459+0000 7f884effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3317675464 0 0) 0x7f8860142710 con 0x7f886014be90 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 10 monclient(hunting): my global_id is 4305 2024-03-25T19:43:29.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.466+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8840004eb0 con 0x7f886014be90 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333798589 0 0) 0x7f8860140d00 con 0x7f88600b4d60 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 10 monclient(hunting): my global_id is 4304 2024-03-25T19:43:29.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.467+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860142710 con 0x7f88600b4d60 2024-03-25T19:43:29.469 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974453976 0 0) 0x7f8860142410 con 0x7f88600b1730 2024-03-25T19:43:29.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 10 monclient(hunting): _negotiate 2024-03-25T19:43:29.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:29.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:29.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 10 monclient(hunting): my global_id is 4306 2024-03-25T19:43:29.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.469+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860140d00 con 0x7f88600b1730 2024-03-25T19:43:29.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2084073188 0 0) 0x7f8860142710 con 0x7f88600b4d60 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8860142410 con 0x7f88600b4d60 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8854001a80 con 0x7f88600b4d60 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 10 monclient(hunting): dump: 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.476 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.477 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.477 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.475+0000 7f884effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778308505 0 0) 0x7f8860140d00 con 0x7f88600b1730 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8860142710 con 0x7f88600b1730 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f88500018f0 con 0x7f88600b1730 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.479+0000 7f884effd640 10 monclient(hunting): dump: 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.480 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 972811079 0 0) 0x7f8840004eb0 con 0x7f886014be90 2024-03-25T19:43:29.481 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8860140d00 con 0x7f886014be90 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8858001e80 con 0x7f886014be90 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.480+0000 7f884effd640 10 monclient(hunting): dump: 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.482 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 602076431 0 0) 0x7f8860142410 con 0x7f88600b4d60 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 >> v1:172.21.15.163:6789/0 conn(0x7f88600b1730 legacy=0x7f88601405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 >> v1:172.21.15.130:6789/0 conn(0x7f886014be90 legacy=0x7f88601e2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 10 monclient: _finish_hunting 0 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 1 monclient: found mon.c 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8860139000 con 0x7f88600b4d60 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 10 monclient: _finish_auth 0 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 10 monclient: _check_auth_tickets 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f884effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 5 monclient: authenticate success, global_id 4304 2024-03-25T19:43:29.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 10 monclient: _renew_subs 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8860141c30 con 0x7f88600b4d60 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 10 monclient: _renew_subs 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88601432b0 con 0x7f88600b4d60 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.486+0000 7f886499d640 1 librados: init done 2024-03-25T19:43:29.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.487+0000 7f886499d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:29.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.487+0000 7f886499d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:29.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.487+0000 7f886499d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:29.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.487+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88601428c0 con 0x7f88600b4d60 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8854001c80 con 0x7f88600b4d60 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f88540038c0 con 0x7f88600b4d60 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f884effd640 10 monclient: dump: 2024-03-25T19:43:29.797 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:29.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.796+0000 7f885cd99640 4 set_mon_vals no callback set 2024-03-25T19:43:29.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.797+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8854003ae0 con 0x7f88600b4d60 2024-03-25T19:43:29.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.798+0000 7f885e59c640 10 monclient: get_auth_request con 0x7f884006e0f0 auth_method 0 2024-03-25T19:43:29.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.798+0000 7f884effd640 10 client.4304.objecter ms_handle_connect 0x7f884006e0f0 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8854004460 con 0x7f88600b4d60 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 10 client.4304.objecter ms_dispatch 0x7f8860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88540b18b0 con 0x7f88600b4d60 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:29.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:29.802+0000 7f884effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:30.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.052+0000 7f886499d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.052+0000 7f886499d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.052+0000 7f886499d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:30.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.052+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88600627b0 con 0x7f88600b4d60 2024-03-25T19:43:30.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.220+0000 7f884effd640 1 -- 172.21.15.130:0/2075503886 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8854050180 con 0x7f88600b4d60 2024-03-25T19:43:30.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.220+0000 7f884effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:30.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.220+0000 7f884effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:30.221 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:30.221 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:30.223 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 10 librados: watch_flush enter 2024-03-25T19:43:30.223 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 10 librados: watch_flush exit 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 10 client.4304.objecter successfully canceled tick 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 >> v1:172.21.15.163:6800/1217380380 conn(0x7f884006e0f0 legacy=0x7f88400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 10 monclient: shutdown 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.222+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 >> v1:172.21.15.130:6790/0 conn(0x7f88600b4d60 legacy=0x7f8860142ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.223+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 shutdown_connections 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.223+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 >> 172.21.15.130:0/2075503886 conn(0x7f88600617a0 msgr2=0x7f88600b6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.223+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 shutdown_connections 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.223+0000 7f886499d640 1 -- 172.21.15.130:0/2075503886 wait complete. 2024-03-25T19:43:30.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:30.223+0000 7f886499d640 1 librados: shutdown 2024-03-25T19:43:30.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:37.042 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:37.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 10 monclient: get_monmap_and_config 2024-03-25T19:43:37.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 10 monclient: build_initial_monmap 2024-03-25T19:43:37.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 10 monclient: monmap: 2024-03-25T19:43:37.160 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:37.160 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:37.161907+0000 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:37.161907+0000 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:37.161 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 1 Processor -- start 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 1 -- start start 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.161+0000 7ffa75330640 10 monclient: init 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:37.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-a con 0x7ffa700a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-b con 0x7ffa7013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-c con 0x7ffa701430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa70142520 con 0x7ffa700a98f0 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700b28a0 con 0x7ffa701430c0 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa700b2a30 con 0x7ffa7013f710 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:37.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.162+0000 7ffa75330640 10 monclient(hunting): authenticate will time out at 5093.308100s 2024-03-25T19:43:37.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.170+0000 7ffa6effd640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ffa700a98f0 0x7ffa700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:55368/0 (socket says 172.21.15.130:55368) 2024-03-25T19:43:37.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.170+0000 7ffa6effd640 1 -- 172.21.15.130:0/2521974309 learned_addr learned my addr 172.21.15.130:0/2521974309 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282278985 0 0) 0x7ffa700b28a0 con 0x7ffa701430c0 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 10 monclient(hunting): my global_id is 4309 2024-03-25T19:43:37.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.184+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa58003520 con 0x7ffa701430c0 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740910297 0 0) 0x7ffa70142520 con 0x7ffa700a98f0 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 10 monclient(hunting): my global_id is 4308 2024-03-25T19:43:37.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.190+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa700b28a0 con 0x7ffa700a98f0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 646879609 0 0) 0x7ffa58003520 con 0x7ffa701430c0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 >> v1:172.21.15.130:6790/0 conn(0x7ffa7013f710 legacy=0x7ffa701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 >> v1:172.21.15.130:6789/0 conn(0x7ffa700a98f0 legacy=0x7ffa700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 monclient: found mon.noname-c 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa700b44c0 con 0x7ffa701430c0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa68001a10 con 0x7ffa701430c0 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa75330640 5 monclient: authenticate success, global_id 4309 2024-03-25T19:43:37.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa75330640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa6dffb640 10 monclient: dump: 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.193+0000 7ffa75330640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa68001b70 con 0x7ffa701430c0 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 1 -- 172.21.15.130:0/2521974309 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa68001f80 con 0x7ffa701430c0 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa75330640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa75330640 4 set_mon_vals no callback set 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa6dffb640 10 monclient: dump: 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.196 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa75330640 10 monclient: shutdown 2024-03-25T19:43:37.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa75330640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:37.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.196+0000 7ffa75330640 1 -- 172.21.15.130:0/2521974309 >> v1:172.21.15.163:6789/0 conn(0x7ffa701430c0 legacy=0x7ffa7014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 1 -- 172.21.15.130:0/2521974309 shutdown_connections 2024-03-25T19:43:37.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 1 -- 172.21.15.130:0/2521974309 >> 172.21.15.130:0/2521974309 conn(0x7ffa700b7ea0 msgr2=0x7ffa700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 1 -- 172.21.15.130:0/2521974309 shutdown_connections 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 1 -- 172.21.15.130:0/2521974309 wait complete. 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 10 monclient: build_initial_monmap 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.197+0000 7ffa75330640 10 monclient: monmap: 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 librados: starting msgr at 2024-03-25T19:43:37.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 librados: starting objecter 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 Processor -- start 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 -- start start 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 librados: setting wanted keys 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 1 librados: calling monclient init 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.198+0000 7ffa75330640 10 monclient: init 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.199+0000 7ffa75330640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:37.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.199+0000 7ffa75330640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-a con 0x7ffa700a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-c con 0x7ffa7013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 10 monclient(hunting): picked mon.noname-b con 0x7ffa701430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa7013c2b0 con 0x7ffa700a98f0 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa70136f10 con 0x7ffa701430c0 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa701370a0 con 0x7ffa7013f710 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:37.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.200+0000 7ffa75330640 10 monclient(hunting): authenticate will time out at 5093.346057s 2024-03-25T19:43:37.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.216+0000 7ffa6f7fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffa701430c0 0x7ffa70133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:47598/0 (socket says 172.21.15.130:47598) 2024-03-25T19:43:37.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.216+0000 7ffa6f7fe640 1 -- 172.21.15.130:0/2931014420 learned_addr learned my addr 172.21.15.130:0/2931014420 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:37.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.221+0000 7ffa577fe640 10 client.?.objecter ms_handle_connect 0x7ffa701430c0 2024-03-25T19:43:37.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.221+0000 7ffa577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:37.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.225+0000 7ffa577fe640 10 client.?.objecter ms_handle_connect 0x7ffa700a98f0 2024-03-25T19:43:37.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.225+0000 7ffa577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:37.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.230+0000 7ffa577fe640 10 client.?.objecter ms_handle_connect 0x7ffa7013f710 2024-03-25T19:43:37.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.230+0000 7ffa577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 503846956 0 0) 0x7ffa701370a0 con 0x7ffa7013f710 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 10 monclient(hunting): my global_id is 4310 2024-03-25T19:43:37.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.234+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa5c004eb0 con 0x7ffa7013f710 2024-03-25T19:43:37.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1084496608 0 0) 0x7ffa70136f10 con 0x7ffa701430c0 2024-03-25T19:43:37.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 10 monclient(hunting): my global_id is 4312 2024-03-25T19:43:37.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.236+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa701370a0 con 0x7ffa701430c0 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2230269340 0 0) 0x7ffa7013c2b0 con 0x7ffa700a98f0 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 10 monclient(hunting): my global_id is 4311 2024-03-25T19:43:37.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.245+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa70136f10 con 0x7ffa700a98f0 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2439817269 0 0) 0x7ffa5c004eb0 con 0x7ffa7013f710 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa7013c2b0 con 0x7ffa7013f710 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa640018c0 con 0x7ffa7013f710 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 10 monclient(hunting): dump: 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.257 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.258 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.257+0000 7ffa577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657217939 0 0) 0x7ffa70136f10 con 0x7ffa700a98f0 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa5c004eb0 con 0x7ffa700a98f0 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa60001930 con 0x7ffa700a98f0 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.262+0000 7ffa577fe640 10 monclient(hunting): dump: 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.261 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.262 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 384020683 0 0) 0x7ffa701370a0 con 0x7ffa701430c0 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa70136f10 con 0x7ffa701430c0 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa68001f40 con 0x7ffa701430c0 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.264+0000 7ffa577fe640 10 monclient(hunting): dump: 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.264 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.265 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.265 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.265 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.265 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3849209345 0 0) 0x7ffa7013c2b0 con 0x7ffa7013f710 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 >> v1:172.21.15.163:6789/0 conn(0x7ffa701430c0 legacy=0x7ffa70133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 >> v1:172.21.15.130:6789/0 conn(0x7ffa700a98f0 legacy=0x7ffa701d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 1 monclient: found mon.c 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:37.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa70137230 con 0x7ffa7013f710 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 5 monclient: authenticate success, global_id 4310 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 10 monclient: _renew_subs 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7013b620 con 0x7ffa7013f710 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 10 client.4310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 10 monclient: _renew_subs 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa70133710 con 0x7ffa7013f710 2024-03-25T19:43:37.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.268+0000 7ffa75330640 1 librados: init done 2024-03-25T19:43:37.270 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.269+0000 7ffa75330640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:37.270 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.269+0000 7ffa75330640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:37.270 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.269+0000 7ffa75330640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:37.270 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.269+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa700627b0 con 0x7ffa7013f710 2024-03-25T19:43:37.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.282+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa640035f0 con 0x7ffa7013f710 2024-03-25T19:43:37.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.282+0000 7ffa577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:37.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.282+0000 7ffa6d7fa640 4 set_mon_vals no callback set 2024-03-25T19:43:37.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.587+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa64003760 con 0x7ffa7013f710 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.587+0000 7ffa577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.587+0000 7ffa577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.587+0000 7ffa577fe640 10 monclient: dump: 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:37.587 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:37.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.588+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ffa64019730 con 0x7ffa7013f710 2024-03-25T19:43:37.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.588+0000 7ffa6effd640 10 monclient: get_auth_request con 0x7ffa5c06e060 auth_method 0 2024-03-25T19:43:37.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.589+0000 7ffa577fe640 10 client.4310.objecter ms_handle_connect 0x7ffa5c06e060 2024-03-25T19:43:37.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ffa640534c0 con 0x7ffa7013f710 2024-03-25T19:43:37.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 10 client.4310.objecter ms_dispatch 0x7ffa70000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:37.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 3 client.4310.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:37.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 3 client.4310.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:37.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 20 client.4310.objecter dump_active .. 0 homeless 2024-03-25T19:43:37.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa6404ee00 con 0x7ffa7013f710 2024-03-25T19:43:37.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:37.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.594+0000 7ffa577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:37.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.847+0000 7ffa75330640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.847+0000 7ffa75330640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.847+0000 7ffa75330640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:43:37.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.847+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa700008d0 con 0x7ffa7013f710 2024-03-25T19:43:37.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.876+0000 7ffa577fe640 1 -- 172.21.15.130:0/2931014420 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ffa64050f00 con 0x7ffa7013f710 2024-03-25T19:43:37.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.876+0000 7ffa577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:37.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.876+0000 7ffa577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:37.876 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:37.876 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 10 librados: watch_flush enter 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 10 librados: watch_flush exit 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 20 client.4310.objecter shutdown clearing up homeless session... 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 10 client.4310.objecter successfully canceled tick 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 >> v1:172.21.15.163:6800/1217380380 conn(0x7ffa5c06e060 legacy=0x7ffa5c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 10 monclient: shutdown 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 >> v1:172.21.15.130:6790/0 conn(0x7ffa7013f710 legacy=0x7ffa701377d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 shutdown_connections 2024-03-25T19:43:37.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 >> 172.21.15.130:0/2931014420 conn(0x7ffa700b7ea0 msgr2=0x7ffa70074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:37.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 shutdown_connections 2024-03-25T19:43:37.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 -- 172.21.15.130:0/2931014420 wait complete. 2024-03-25T19:43:37.880 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:37.879+0000 7ffa75330640 1 librados: shutdown 2024-03-25T19:43:37.892 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:39.644 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:24.577 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.659 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:24.592 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.690 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:24.624 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.695 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:24.596 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.704 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:24.605 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.711 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:24.644 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.713 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:24.646 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.734 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:24.635 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:39.739 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:24.672 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:44.698 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.813+0000 7f6ea72f1640 10 monclient: get_monmap_and_config 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.813+0000 7f6ea72f1640 10 monclient: build_initial_monmap 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.813+0000 7f6ea72f1640 10 monclient: monmap: 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:44.813 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:44.814626+0000 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:44.814626+0000 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:44.814 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:44.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.814+0000 7f6ea72f1640 1 Processor -- start 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.814+0000 7f6ea72f1640 1 -- start start 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.814+0000 7f6ea72f1640 10 monclient: init 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0142520 con 0x7f6ea013f710 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00b28a0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00b2a30 con 0x7f6ea01430c0 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:44.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.815+0000 7f6ea72f1640 10 monclient(hunting): authenticate will time out at 5100.961022s 2024-03-25T19:43:44.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.821+0000 7f6ea508d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6ea00a98f0 0x7f6ea00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:52114/0 (socket says 172.21.15.130:52114) 2024-03-25T19:43:44.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.821+0000 7f6ea508d640 1 -- 172.21.15.130:0/2828334277 learned_addr learned my addr 172.21.15.130:0/2828334277 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:44.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965539181 0 0) 0x7f6ea0142520 con 0x7f6ea013f710 2024-03-25T19:43:44.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 10 monclient(hunting): my global_id is 4314 2024-03-25T19:43:44.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.830+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e8c003520 con 0x7f6ea013f710 2024-03-25T19:43:44.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301669437 0 0) 0x7f6ea00b28a0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 10 monclient(hunting): my global_id is 4315 2024-03-25T19:43:44.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.835+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea0142520 con 0x7f6ea00a98f0 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1598518764 0 0) 0x7f6ea00b2a30 con 0x7f6ea01430c0 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 10 monclient(hunting): my global_id is 4313 2024-03-25T19:43:44.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.836+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea00b28a0 con 0x7f6ea01430c0 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 652139575 0 0) 0x7f6ea0142520 con 0x7f6ea00a98f0 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 >> v1:172.21.15.130:6790/0 conn(0x7f6ea01430c0 legacy=0x7f6ea014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 >> v1:172.21.15.130:6789/0 conn(0x7f6ea013f710 legacy=0x7f6ea01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: _finish_hunting 0 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 monclient: found mon.noname-c 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:44.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00b44c0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: _finish_auth 0 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: _check_auth_tickets 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e9c001670 con 0x7f6ea00a98f0 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6e97fff640 10 monclient: dump: 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:44.846 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:44.847 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:44.847 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:44.847 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:44.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6ea72f1640 5 monclient: authenticate success, global_id 4315 2024-03-25T19:43:44.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.845+0000 7f6ea72f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e9c0017d0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 1 -- 172.21.15.130:0/2828334277 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e9c001be0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 4 set_mon_vals no callback set 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6e97fff640 10 monclient: dump: 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:44.859 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:44.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 10 monclient: shutdown 2024-03-25T19:43:44.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2828334277 >> v1:172.21.15.163:6789/0 conn(0x7f6ea00a98f0 legacy=0x7f6ea00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2828334277 shutdown_connections 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.859+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2828334277 >> 172.21.15.130:0/2828334277 conn(0x7f6ea00b7ea0 msgr2=0x7f6ea00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2828334277 shutdown_connections 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2828334277 wait complete. 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 10 monclient: build_initial_monmap 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 10 monclient: monmap: 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:44.861 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 1 librados: starting msgr at 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.860+0000 7f6ea72f1640 1 librados: starting objecter 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 1 Processor -- start 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 1 -- start start 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 1 librados: setting wanted keys 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 1 librados: calling monclient init 2024-03-25T19:43:44.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 10 monclient: init 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6ea00a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.861+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6ea013f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6ea01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0137f50 con 0x7f6ea00a98f0 2024-03-25T19:43:44.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea0139830 con 0x7f6ea013f710 2024-03-25T19:43:44.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ea00aba50 con 0x7f6ea01430c0 2024-03-25T19:43:44.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:44.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.862+0000 7f6ea72f1640 10 monclient(hunting): authenticate will time out at 5101.007756s 2024-03-25T19:43:44.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.868+0000 7f6ea508d640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f6ea00a98f0 0x7f6ea0134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:59912/0 (socket says 172.21.15.130:59912) 2024-03-25T19:43:44.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.868+0000 7f6ea508d640 1 -- 172.21.15.130:0/2821281396 learned_addr learned my addr 172.21.15.130:0/2821281396 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:44.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.869+0000 7f6e95ffb640 10 client.?.objecter ms_handle_connect 0x7f6ea00a98f0 2024-03-25T19:43:44.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.869+0000 7f6e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893378309 0 0) 0x7f6ea0137f50 con 0x7f6ea00a98f0 2024-03-25T19:43:44.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 10 monclient(hunting): my global_id is 4317 2024-03-25T19:43:44.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.874+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.879+0000 7f6e95ffb640 10 client.?.objecter ms_handle_connect 0x7f6ea01430c0 2024-03-25T19:43:44.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.879+0000 7f6e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.879+0000 7f6e95ffb640 10 client.?.objecter ms_handle_connect 0x7f6ea013f710 2024-03-25T19:43:44.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.879+0000 7f6e95ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:44.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129612426 0 0) 0x7f6ea00aba50 con 0x7f6ea01430c0 2024-03-25T19:43:44.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 10 monclient(hunting): my global_id is 4316 2024-03-25T19:43:44.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.892+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea0137f50 con 0x7f6ea01430c0 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 158086635 0 0) 0x7f6e84004eb0 con 0x7f6ea00a98f0 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ea00aba50 con 0x7f6ea00a98f0 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e9c001c20 con 0x7f6ea00a98f0 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:43:44.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 10 monclient(hunting): dump: 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:44.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.893+0000 7f6e95ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:44.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898788344 0 0) 0x7f6ea0139830 con 0x7f6ea013f710 2024-03-25T19:43:44.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:43:44.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:44.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:44.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 10 monclient(hunting): my global_id is 4318 2024-03-25T19:43:44.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.902+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e84004eb0 con 0x7f6ea013f710 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4126284604 0 0) 0x7f6ea00aba50 con 0x7f6ea00a98f0 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 >> v1:172.21.15.130:6790/0 conn(0x7f6ea01430c0 legacy=0x7f6ea0136240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 >> v1:172.21.15.163:6789/0 conn(0x7f6ea013f710 legacy=0x7f6ea0135b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 1 monclient: found mon.a 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea00abd30 con 0x7f6ea00a98f0 2024-03-25T19:43:44.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 10 monclient: _finish_auth 0 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6e95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 5 monclient: authenticate success, global_id 4317 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 10 monclient: _renew_subs 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ea0137a00 con 0x7f6ea00a98f0 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 10 monclient: _renew_subs 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ea0134970 con 0x7f6ea00a98f0 2024-03-25T19:43:44.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.907+0000 7f6ea72f1640 1 librados: init done 2024-03-25T19:43:44.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.908+0000 7f6ea72f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.909+0000 7f6ea72f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:44.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.909+0000 7f6ea72f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:44.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:44.909+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ea00627b0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e9c004a90 con 0x7f6ea00a98f0 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e9c004de0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e95ffb640 10 monclient: dump: 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:45.209 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.209+0000 7f6e977fe640 4 set_mon_vals no callback set 2024-03-25T19:43:45.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.210+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f6e9c005000 con 0x7f6ea00a98f0 2024-03-25T19:43:45.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.210+0000 7f6ea488c640 10 monclient: get_auth_request con 0x7f6e8406dd60 auth_method 0 2024-03-25T19:43:45.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.211+0000 7f6e95ffb640 10 client.4317.objecter ms_handle_connect 0x7f6e8406dd60 2024-03-25T19:43:45.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f6e9c0549b0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 10 client.4317.objecter ms_dispatch 0x7f6ea0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 3 client.4317.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 3 client.4317.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 20 client.4317.objecter dump_active .. 0 homeless 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e9c0504c0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:45.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.217+0000 7f6e95ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:45.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.434+0000 7f6ea72f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:45.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.434+0000 7f6ea72f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:45.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.434+0000 7f6ea72f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:45.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.434+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ea00b71a0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.468+0000 7f6e95ffb640 1 -- 172.21.15.130:0/2821281396 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f6e9c0523f0 con 0x7f6ea00a98f0 2024-03-25T19:43:45.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.468+0000 7f6e95ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:45.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.468+0000 7f6e95ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:45.468 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:45.468 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 10 librados: watch_flush enter 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 10 librados: watch_flush exit 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 20 client.4317.objecter shutdown clearing up homeless session... 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 10 client.4317.objecter successfully canceled tick 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 >> v1:172.21.15.163:6800/1217380380 conn(0x7f6e8406dd60 legacy=0x7f6e84070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 10 monclient: shutdown 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:45.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 >> v1:172.21.15.130:6789/0 conn(0x7f6ea00a98f0 legacy=0x7f6ea0134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:45.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 shutdown_connections 2024-03-25T19:43:45.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 >> 172.21.15.130:0/2821281396 conn(0x7f6ea00b7ea0 msgr2=0x7f6ea0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:45.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 shutdown_connections 2024-03-25T19:43:45.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 -- 172.21.15.130:0/2821281396 wait complete. 2024-03-25T19:43:45.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:45.471+0000 7f6ea72f1640 1 librados: shutdown 2024-03-25T19:43:45.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:52.290 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.406+0000 7fa99eb5d640 10 monclient: get_monmap_and_config 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.406+0000 7fa99eb5d640 10 monclient: build_initial_monmap 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.406+0000 7fa99eb5d640 10 monclient: monmap: 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:43:52.407634+0000 2024-03-25T19:43:52.406 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:43:52.407634+0000 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:43:52.407 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.407+0000 7fa99eb5d640 1 Processor -- start 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.407+0000 7fa99eb5d640 1 -- start start 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.407+0000 7fa99eb5d640 10 monclient: init 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9980b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa9980b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa99814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:52.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980af940 con 0x7fa9980b1730 2024-03-25T19:43:52.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b0a70 con 0x7fa9980b4d60 2024-03-25T19:43:52.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980b0d50 con 0x7fa99814be90 2024-03-25T19:43:52.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:52.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.408+0000 7fa99eb5d640 10 monclient(hunting): authenticate will time out at 5108.553937s 2024-03-25T19:43:52.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.421+0000 7fa99d0fa640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fa99814be90 0x7fa99814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51020/0 (socket says 172.21.15.130:51020) 2024-03-25T19:43:52.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.421+0000 7fa99d0fa640 1 -- 172.21.15.130:0/1498379739 learned_addr learned my addr 172.21.15.130:0/1498379739 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:52.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.433+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234193749 0 0) 0x7fa9980b0d50 con 0x7fa99814be90 2024-03-25T19:43:52.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.434+0000 7fa9977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:52.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.434+0000 7fa9977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:52.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.434+0000 7fa9977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:52.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.434+0000 7fa9977fe640 10 monclient(hunting): my global_id is 4319 2024-03-25T19:43:52.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.434+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa988003520 con 0x7fa99814be90 2024-03-25T19:43:52.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894829766 0 0) 0x7fa9980b0a70 con 0x7fa9980b4d60 2024-03-25T19:43:52.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 10 monclient(hunting): _negotiate 2024-03-25T19:43:52.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:52.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:52.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 10 monclient(hunting): my global_id is 4321 2024-03-25T19:43:52.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.441+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9980b0d50 con 0x7fa9980b4d60 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1487118574 0 0) 0x7fa988003520 con 0x7fa99814be90 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 >> v1:172.21.15.163:6789/0 conn(0x7fa9980b4d60 legacy=0x7fa99813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 >> v1:172.21.15.130:6789/0 conn(0x7fa9980b1730 legacy=0x7fa9980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: _finish_hunting 0 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 monclient: found mon.noname-b 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:52.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9980b1030 con 0x7fa99814be90 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: _finish_auth 0 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: _check_auth_tickets 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa990001a10 con 0x7fa99814be90 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa9977fe640 10 monclient: dump: 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.448 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:52.449 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:52.449 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:52.449 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa99eb5d640 5 monclient: authenticate success, global_id 4319 2024-03-25T19:43:52.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.447+0000 7fa99eb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:43:52.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa990001b70 con 0x7fa99814be90 2024-03-25T19:43:52.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 1 -- 172.21.15.130:0/1498379739 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa990001f80 con 0x7fa99814be90 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa9977fe640 10 monclient: dump: 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa99eb5d640 10 monclient: get_monmap_and_config success 2024-03-25T19:43:52.462 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa99eb5d640 4 set_mon_vals no callback set 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa99eb5d640 10 monclient: shutdown 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa99eb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.461+0000 7fa99eb5d640 1 -- 172.21.15.130:0/1498379739 >> v1:172.21.15.130:6790/0 conn(0x7fa99814be90 legacy=0x7fa99814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 1 -- 172.21.15.130:0/1498379739 shutdown_connections 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 1 -- 172.21.15.130:0/1498379739 >> 172.21.15.130:0/1498379739 conn(0x7fa9980617a0 msgr2=0x7fa998061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 1 -- 172.21.15.130:0/1498379739 shutdown_connections 2024-03-25T19:43:52.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 1 -- 172.21.15.130:0/1498379739 wait complete. 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 10 monclient: build_initial_monmap 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.462+0000 7fa99eb5d640 10 monclient: monmap: 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 librados: starting msgr at 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 librados: starting objecter 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 Processor -- start 2024-03-25T19:43:52.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 -- start start 2024-03-25T19:43:52.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 librados: setting wanted keys 2024-03-25T19:43:52.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 1 librados: calling monclient init 2024-03-25T19:43:52.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.463+0000 7fa99eb5d640 10 monclient: init 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient: _reopen_session rank -1 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fa9980b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fa9980b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fa99814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa998137b10 con 0x7fa99814be90 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9981377e0 con 0x7fa9980b1730 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa998137fa0 con 0x7fa9980b4d60 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient(hunting): _renew_subs 2024-03-25T19:43:52.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.464+0000 7fa99eb5d640 10 monclient(hunting): authenticate will time out at 5108.610219s 2024-03-25T19:43:52.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.472+0000 7fa997fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fa9980b4d60 0x7fa9981daed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51042/0 (socket says 172.21.15.130:51042) 2024-03-25T19:43:52.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.472+0000 7fa997fff640 1 -- 172.21.15.130:0/313417186 learned_addr learned my addr 172.21.15.130:0/313417186 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:43:52.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.473+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa99814be90 2024-03-25T19:43:52.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.473+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:52.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.475+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa9980b4d60 2024-03-25T19:43:52.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.475+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:52.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.476+0000 7fa9957fa640 10 client.?.objecter ms_handle_connect 0x7fa9980b1730 2024-03-25T19:43:52.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.476+0000 7fa9957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624766180 0 0) 0x7fa998137b10 con 0x7fa99814be90 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4323 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa97c004eb0 con 0x7fa99814be90 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2072927329 0 0) 0x7fa9981377e0 con 0x7fa9980b1730 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:52.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4324 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.488+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa998137b10 con 0x7fa9980b1730 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808537270 0 0) 0x7fa998137fa0 con 0x7fa9980b4d60 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 10 monclient(hunting): _negotiate 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 10 monclient(hunting): my global_id is 4322 2024-03-25T19:43:52.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.489+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9981377e0 con 0x7fa9980b4d60 2024-03-25T19:43:52.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4046785042 0 0) 0x7fa998137b10 con 0x7fa9980b1730 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa998137fa0 con 0x7fa9980b1730 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa98c001930 con 0x7fa9980b1730 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 10 monclient(hunting): dump: 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:52.503 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.503+0000 7fa9957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3796718492 0 0) 0x7fa97c004eb0 con 0x7fa99814be90 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa998137b10 con 0x7fa99814be90 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa990001e80 con 0x7fa99814be90 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:52.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.504+0000 7fa9957fa640 10 monclient(hunting): dump: 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:52.505 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3974918591 0 0) 0x7fa998137b10 con 0x7fa99814be90 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 >> v1:172.21.15.130:6790/0 conn(0x7fa9980b4d60 legacy=0x7fa9981daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 >> v1:172.21.15.163:6789/0 conn(0x7fa9980b1730 legacy=0x7fa998134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 10 monclient: _finish_hunting 0 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 1 monclient: found mon.a 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:52.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa998142030 con 0x7fa99814be90 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 10 monclient: _finish_auth 0 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 10 monclient: _check_auth_tickets 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa9957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 5 monclient: authenticate success, global_id 4323 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 10 monclient: _renew_subs 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa998140840 con 0x7fa99814be90 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 10 monclient: _renew_subs 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9981357a0 con 0x7fa99814be90 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.518+0000 7fa99eb5d640 1 librados: init done 2024-03-25T19:43:52.519 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.519+0000 7fa99eb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:52.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.519+0000 7fa99eb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:52.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.519+0000 7fa99eb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:52.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.519+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9980627b0 con 0x7fa99814be90 2024-03-25T19:43:52.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa990002020 con 0x7fa99814be90 2024-03-25T19:43:52.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:43:52.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa990004fb0 con 0x7fa99814be90 2024-03-25T19:43:52.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa9957fa640 10 monclient: dump: 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:43:52.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.840+0000 7fa996ffd640 4 set_mon_vals no callback set 2024-03-25T19:43:52.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.841+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fa99001af80 con 0x7fa99814be90 2024-03-25T19:43:52.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.841+0000 7fa99c8f9640 10 monclient: get_auth_request con 0x7fa97c06dfb0 auth_method 0 2024-03-25T19:43:52.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.841+0000 7fa9957fa640 10 client.4323.objecter ms_handle_connect 0x7fa97c06dfb0 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fa990054a40 con 0x7fa99814be90 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 10 client.4323.objecter ms_dispatch 0x7fa998000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa990050550 con 0x7fa99814be90 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:43:52.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:52.846+0000 7fa9957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:43:53.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.100+0000 7fa99eb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.100+0000 7fa99eb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.100+0000 7fa99eb5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:43:53.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.100+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa998137cc0 con 0x7fa99814be90 2024-03-25T19:43:53.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.130+0000 7fa9957fa640 1 -- 172.21.15.130:0/313417186 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fa990052480 con 0x7fa99814be90 2024-03-25T19:43:53.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.130+0000 7fa9957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:43:53.130 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.130+0000 7fa9957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:43:53.130 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:43:53.130 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 10 librados: watch_flush enter 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 10 librados: watch_flush exit 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 10 client.4323.objecter successfully canceled tick 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 >> v1:172.21.15.163:6800/1217380380 conn(0x7fa97c06dfb0 legacy=0x7fa97c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 10 monclient: shutdown 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:43:53.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 >> v1:172.21.15.130:6789/0 conn(0x7fa99814be90 legacy=0x7fa9981e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:43:53.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 shutdown_connections 2024-03-25T19:43:53.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 >> 172.21.15.130:0/313417186 conn(0x7fa9980617a0 msgr2=0x7fa99813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:43:53.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 shutdown_connections 2024-03-25T19:43:53.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 -- 172.21.15.130:0/313417186 wait complete. 2024-03-25T19:43:53.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:43:53.132+0000 7fa99eb5d640 1 librados: shutdown 2024-03-25T19:43:53.143 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:43:59.664 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:44.598 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.712 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:44.645 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.761 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:02:44.694 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.767 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:44.668 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.772 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:44.673 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.782 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:44.715 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.793 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:44.724 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.803 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:02:44.704 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.817 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:02:44.750 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:43:59.948 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:00.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.063+0000 7f5ad2182640 10 monclient: get_monmap_and_config 2024-03-25T19:44:00.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.063+0000 7f5ad2182640 10 monclient: build_initial_monmap 2024-03-25T19:44:00.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.064+0000 7f5ad2182640 10 monclient: monmap: 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:00.064834+0000 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:00.064834+0000 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:00.064 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.064+0000 7f5ad2182640 1 Processor -- start 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 1 -- start start 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 10 monclient: init 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-b con 0x7f5acc0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.065+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-c con 0x7f5acc13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-a con 0x7f5acc1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc142520 con 0x7f5acc1430c0 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc0b28a0 con 0x7f5acc13f710 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc0b2a30 con 0x7f5acc0a98f0 2024-03-25T19:44:00.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:00.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.066+0000 7f5ad2182640 10 monclient(hunting): authenticate will time out at 5116.211799s 2024-03-25T19:44:00.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.080+0000 7f5acb7fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f5acc0a98f0 0x7f5acc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:44290/0 (socket says 172.21.15.130:44290) 2024-03-25T19:44:00.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.080+0000 7f5acb7fe640 1 -- 172.21.15.130:0/2718243280 learned_addr learned my addr 172.21.15.130:0/2718243280 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293506201 0 0) 0x7f5acc0b28a0 con 0x7f5acc13f710 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 10 monclient(hunting): my global_id is 4327 2024-03-25T19:44:00.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.090+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab4003520 con 0x7f5acc13f710 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893798586 0 0) 0x7f5acc0b2a30 con 0x7f5acc0a98f0 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 10 monclient(hunting): my global_id is 4325 2024-03-25T19:44:00.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.093+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5acc0b28a0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 961812432 0 0) 0x7f5acc142520 con 0x7f5acc1430c0 2024-03-25T19:44:00.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient(hunting): my global_id is 4326 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5acc0b2a30 con 0x7f5acc1430c0 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3945219525 0 0) 0x7f5ab4003520 con 0x7f5acc13f710 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 >> v1:172.21.15.130:6790/0 conn(0x7f5acc0a98f0 legacy=0x7f5acc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 >> v1:172.21.15.130:6789/0 conn(0x7f5acc1430c0 legacy=0x7f5acc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:44:00.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 monclient: found mon.noname-c 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0b44c0 con 0x7f5acc13f710 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: _finish_auth 0 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5ac0001690 con 0x7f5acc13f710 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5aca7fc640 10 monclient: dump: 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:00.098 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5ad2182640 5 monclient: authenticate success, global_id 4327 2024-03-25T19:44:00.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.097+0000 7f5ad2182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ac0001870 con 0x7f5acc13f710 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 1 -- 172.21.15.130:0/2718243280 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5ac0001c80 con 0x7f5acc13f710 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5aca7fc640 10 monclient: dump: 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:00.115 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5ad2182640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:00.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.115+0000 7f5ad2182640 4 set_mon_vals no callback set 2024-03-25T19:44:00.117 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 10 monclient: shutdown 2024-03-25T19:44:00.117 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 1 -- 172.21.15.130:0/2718243280 >> v1:172.21.15.163:6789/0 conn(0x7f5acc13f710 legacy=0x7f5acc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 1 -- 172.21.15.130:0/2718243280 shutdown_connections 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 1 -- 172.21.15.130:0/2718243280 >> 172.21.15.130:0/2718243280 conn(0x7f5acc0b7ea0 msgr2=0x7f5acc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 1 -- 172.21.15.130:0/2718243280 shutdown_connections 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.116+0000 7f5ad2182640 1 -- 172.21.15.130:0/2718243280 wait complete. 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.117+0000 7f5ad2182640 10 monclient: build_initial_monmap 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.117+0000 7f5ad2182640 10 monclient: monmap: 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:00.118 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 librados: starting msgr at 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 librados: starting objecter 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 Processor -- start 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 -- start start 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 librados: setting wanted keys 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 1 librados: calling monclient init 2024-03-25T19:44:00.119 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.118+0000 7f5ad2182640 10 monclient: init 2024-03-25T19:44:00.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:00.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:00.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-b con 0x7f5acc0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:00.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-c con 0x7f5acc13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient(hunting): picked mon.noname-a con 0x7f5acc1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc138250 con 0x7f5acc1430c0 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc137d50 con 0x7f5acc0a98f0 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5acc137ee0 con 0x7f5acc13f710 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:00.121 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.119+0000 7f5ad2182640 10 monclient(hunting): authenticate will time out at 5116.264868s 2024-03-25T19:44:00.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.125+0000 7f5acb7fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f5acc0a98f0 0x7f5acc137640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:49832/0 (socket says 172.21.15.130:49832) 2024-03-25T19:44:00.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.125+0000 7f5acb7fe640 1 -- 172.21.15.130:0/2265174528 learned_addr learned my addr 172.21.15.130:0/2265174528 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:00.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.125+0000 7f5ab3fff640 10 client.?.objecter ms_handle_connect 0x7f5acc0a98f0 2024-03-25T19:44:00.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.125+0000 7f5ab3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:00.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212555419 0 0) 0x7f5acc137d50 con 0x7f5acc0a98f0 2024-03-25T19:44:00.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 10 monclient(hunting): my global_id is 4330 2024-03-25T19:44:00.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.133+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8004eb0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.141+0000 7f5ab3fff640 10 client.?.objecter ms_handle_connect 0x7f5acc13f710 2024-03-25T19:44:00.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.141+0000 7f5ab3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:00.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.143+0000 7f5ab3fff640 10 client.?.objecter ms_handle_connect 0x7f5acc1430c0 2024-03-25T19:44:00.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.143+0000 7f5ab3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268900461 0 0) 0x7f5acc138250 con 0x7f5acc1430c0 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 10 monclient(hunting): my global_id is 4329 2024-03-25T19:44:00.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.148+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5acc137d50 con 0x7f5acc1430c0 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 406289741 0 0) 0x7f5ab8004eb0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5acc138250 con 0x7f5acc0a98f0 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5abc001970 con 0x7f5acc0a98f0 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): dump: 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391196431 0 0) 0x7f5acc137ee0 con 0x7f5acc13f710 2024-03-25T19:44:00.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:00.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:00.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:00.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 10 monclient(hunting): my global_id is 4328 2024-03-25T19:44:00.151 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.149+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ab8004eb0 con 0x7f5acc13f710 2024-03-25T19:44:00.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109818713 0 0) 0x7f5acc138250 con 0x7f5acc0a98f0 2024-03-25T19:44:00.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 >> v1:172.21.15.130:6790/0 conn(0x7f5acc13f710 legacy=0x7f5acc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 >> v1:172.21.15.130:6789/0 conn(0x7f5acc1430c0 legacy=0x7f5acc1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 1 monclient: found mon.b 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc0b7730 con 0x7f5acc0a98f0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 10 monclient: _finish_auth 0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ab3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 5 monclient: authenticate success, global_id 4330 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 10 monclient: _renew_subs 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5acc0b7920 con 0x7f5acc0a98f0 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:00.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 10 monclient: _renew_subs 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5acc1e33d0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.163+0000 7f5ad2182640 1 librados: init done 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.164+0000 7f5ad2182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.164+0000 7f5ad2182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.164+0000 7f5ad2182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.164+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5acc0627b0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5abc0047e0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5abc004af0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:00.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ab3fff640 10 monclient: dump: 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:00.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.446+0000 7f5ac9ffb640 4 set_mon_vals no callback set 2024-03-25T19:44:00.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.447+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f5abc01aac0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.448+0000 7f5acaffd640 10 monclient: get_auth_request con 0x7f5ab806ddb0 auth_method 0 2024-03-25T19:44:00.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.448+0000 7f5ab3fff640 10 client.4330.objecter ms_handle_connect 0x7f5ab806ddb0 2024-03-25T19:44:00.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.455+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f5abc054870 con 0x7f5acc0a98f0 2024-03-25T19:44:00.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.455+0000 7f5ab3fff640 10 client.4330.objecter ms_dispatch 0x7f5acc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:00.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.455+0000 7f5ab3fff640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:00.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.455+0000 7f5ab3fff640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:00.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.455+0000 7f5ab3fff640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T19:44:00.456 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.456+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5abc0501b0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.456 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.456+0000 7f5ab3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:00.456 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.456+0000 7f5ab3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:00.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.711+0000 7f5ad2182640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.711+0000 7f5ad2182640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.711+0000 7f5ad2182640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:00.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.711+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5acc13df50 con 0x7f5acc0a98f0 2024-03-25T19:44:00.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.858+0000 7f5ab3fff640 1 -- 172.21.15.130:0/2265174528 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f5abc0522b0 con 0x7f5acc0a98f0 2024-03-25T19:44:00.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.858+0000 7f5ab3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:00.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.858+0000 7f5ab3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:00.858 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:00.858 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:00.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.859+0000 7f5ad2182640 10 librados: watch_flush enter 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 10 librados: watch_flush exit 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 10 client.4330.objecter successfully canceled tick 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 >> v1:172.21.15.163:6800/1217380380 conn(0x7f5ab806ddb0 legacy=0x7f5ab8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 10 monclient: shutdown 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 >> v1:172.21.15.163:6789/0 conn(0x7f5acc0a98f0 legacy=0x7f5acc137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 shutdown_connections 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 >> 172.21.15.130:0/2265174528 conn(0x7f5acc0b7ea0 msgr2=0x7f5acc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 shutdown_connections 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 -- 172.21.15.130:0/2265174528 wait complete. 2024-03-25T19:44:00.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:00.860+0000 7f5ad2182640 1 librados: shutdown 2024-03-25T19:44:00.870 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:07.677 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:07.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.792+0000 7f36b0863640 10 monclient: get_monmap_and_config 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.792+0000 7f36b0863640 10 monclient: build_initial_monmap 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.792+0000 7f36b0863640 10 monclient: monmap: 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:07.793182+0000 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:07.793182+0000 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:07.792 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.793+0000 7f36b0863640 1 Processor -- start 2024-03-25T19:44:07.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.793+0000 7f36b0863640 1 -- start start 2024-03-25T19:44:07.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.793+0000 7f36b0863640 10 monclient: init 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-c con 0x7f36ac0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-b con 0x7f36ac13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-a con 0x7f36ac1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac142520 con 0x7f36ac1430c0 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac0b28a0 con 0x7f36ac0a98f0 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac0b2a30 con 0x7f36ac13f710 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:07.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.794+0000 7f36b0863640 10 monclient(hunting): authenticate will time out at 5123.940185s 2024-03-25T19:44:07.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.809+0000 7f36a9d9b640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f36ac13f710 0x7f36ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:36986/0 (socket says 172.21.15.130:36986) 2024-03-25T19:44:07.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.809+0000 7f36a9d9b640 1 -- 172.21.15.130:0/3520398677 learned_addr learned my addr 172.21.15.130:0/3520398677 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909382391 0 0) 0x7f36ac0b2a30 con 0x7f36ac13f710 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 10 monclient(hunting): my global_id is 4331 2024-03-25T19:44:07.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.819+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3694003520 con 0x7f36ac13f710 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716789681 0 0) 0x7f36ac142520 con 0x7f36ac1430c0 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 10 monclient(hunting): my global_id is 4332 2024-03-25T19:44:07.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.822+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac0b2a30 con 0x7f36ac1430c0 2024-03-25T19:44:07.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1641956429 0 0) 0x7f3694003520 con 0x7f36ac13f710 2024-03-25T19:44:07.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 >> v1:172.21.15.163:6789/0 conn(0x7f36ac0a98f0 legacy=0x7f36ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 >> v1:172.21.15.130:6789/0 conn(0x7f36ac1430c0 legacy=0x7f36ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 monclient: found mon.noname-b 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac0b44c0 con 0x7f36ac13f710 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 10 monclient: _finish_auth 0 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:07.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f36a0001690 con 0x7f36ac13f710 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.828+0000 7f36b0863640 5 monclient: authenticate success, global_id 4331 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.829+0000 7f36b0863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.829+0000 7f36a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.829+0000 7f36a959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.829+0000 7f36a959a640 10 monclient: dump: 2024-03-25T19:44:07.831 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:07.832 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:07.833 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.829+0000 7f36b0863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:07.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.842+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a00017f0 con 0x7f36ac13f710 2024-03-25T19:44:07.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36a959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:07.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36a959a640 1 -- 172.21.15.130:0/3520398677 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f36a0001c00 con 0x7f36ac13f710 2024-03-25T19:44:07.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36b0863640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:07.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36b0863640 4 set_mon_vals no callback set 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36a959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36a959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36a959a640 10 monclient: dump: 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:07.843 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36b0863640 10 monclient: shutdown 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36b0863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.843+0000 7f36b0863640 1 -- 172.21.15.130:0/3520398677 >> v1:172.21.15.130:6790/0 conn(0x7f36ac13f710 legacy=0x7f36ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.844+0000 7f36b0863640 1 -- 172.21.15.130:0/3520398677 shutdown_connections 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.844+0000 7f36b0863640 1 -- 172.21.15.130:0/3520398677 >> 172.21.15.130:0/3520398677 conn(0x7f36ac0b7ea0 msgr2=0x7f36ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.844+0000 7f36b0863640 1 -- 172.21.15.130:0/3520398677 shutdown_connections 2024-03-25T19:44:07.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.844+0000 7f36b0863640 1 -- 172.21.15.130:0/3520398677 wait complete. 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 10 monclient: build_initial_monmap 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 10 monclient: monmap: 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:07.846 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 librados: starting msgr at 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 librados: starting objecter 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 Processor -- start 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 -- start start 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 librados: setting wanted keys 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 1 librados: calling monclient init 2024-03-25T19:44:07.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.845+0000 7f36b0863640 10 monclient: init 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-a con 0x7f36ac0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-c con 0x7f36ac13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient(hunting): picked mon.noname-b con 0x7f36ac1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac0aba00 con 0x7f36ac0a98f0 2024-03-25T19:44:07.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac1d7490 con 0x7f36ac1430c0 2024-03-25T19:44:07.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36ac1d8360 con 0x7f36ac13f710 2024-03-25T19:44:07.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:07.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.846+0000 7f36b0863640 10 monclient(hunting): authenticate will time out at 5123.992279s 2024-03-25T19:44:07.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.860+0000 7f36aad9d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f36ac1430c0 0x7f36ac133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:59484/0 (socket says 172.21.15.130:59484) 2024-03-25T19:44:07.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.860+0000 7f36aad9d640 1 -- 172.21.15.130:0/1169758254 learned_addr learned my addr 172.21.15.130:0/1169758254 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:07.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.862+0000 7f369affd640 10 client.?.objecter ms_handle_connect 0x7f36ac1430c0 2024-03-25T19:44:07.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.862+0000 7f369affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.865+0000 7f369affd640 10 client.?.objecter ms_handle_connect 0x7f36ac13f710 2024-03-25T19:44:07.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.865+0000 7f369affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728588056 0 0) 0x7f36ac1d8360 con 0x7f36ac13f710 2024-03-25T19:44:07.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.876 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 monclient(hunting): my global_id is 4334 2024-03-25T19:44:07.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f368c004eb0 con 0x7f36ac13f710 2024-03-25T19:44:07.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 client.?.objecter ms_handle_connect 0x7f36ac0a98f0 2024-03-25T19:44:07.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.877+0000 7f369affd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514376036 0 0) 0x7f36ac1d7490 con 0x7f36ac1430c0 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 10 monclient(hunting): my global_id is 4336 2024-03-25T19:44:07.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.879+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac1d8360 con 0x7f36ac1430c0 2024-03-25T19:44:07.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4029204106 0 0) 0x7f36ac1d8360 con 0x7f36ac1430c0 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ac1d7490 con 0x7f36ac1430c0 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f36a4001c10 con 0x7f36ac1430c0 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 10 monclient(hunting): dump: 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:07.889 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:07.890 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:07.890 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:07.890 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.889+0000 7f369affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:07.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1445409884 0 0) 0x7f368c004eb0 con 0x7f36ac13f710 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36ac1d8360 con 0x7f36ac13f710 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f36a0001bc0 con 0x7f36ac13f710 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.890+0000 7f369affd640 10 monclient(hunting): dump: 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:07.891 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:07.892 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:07.892 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:07.892 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:07.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.899+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2179457510 0 0) 0x7f36ac1d7490 con 0x7f36ac1430c0 2024-03-25T19:44:07.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.899+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 >> v1:172.21.15.130:6790/0 conn(0x7f36ac13f710 legacy=0x7f36ac1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 >> v1:172.21.15.130:6789/0 conn(0x7f36ac0a98f0 legacy=0x7f36ac1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 1 monclient: found mon.b 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36ac0abc30 con 0x7f36ac1430c0 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 10 monclient: _finish_auth 0 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f369affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 5 monclient: authenticate success, global_id 4336 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 10 monclient: _renew_subs 2024-03-25T19:44:07.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36ac137940 con 0x7f36ac1430c0 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 10 monclient: _renew_subs 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36ac135220 con 0x7f36ac1430c0 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.900+0000 7f36b0863640 1 librados: init done 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.901+0000 7f36b0863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.901+0000 7f36b0863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.901+0000 7f36b0863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:07.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:07.901+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36ac0627b0 con 0x7f36ac1430c0 2024-03-25T19:44:08.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a4003990 con 0x7f36ac1430c0 2024-03-25T19:44:08.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:08.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f36a4002100 con 0x7f36ac1430c0 2024-03-25T19:44:08.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f369affd640 10 monclient: dump: 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.187+0000 7f36a8d99640 4 set_mon_vals no callback set 2024-03-25T19:44:08.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.188+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f36a4003da0 con 0x7f36ac1430c0 2024-03-25T19:44:08.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.189+0000 7f36aa59c640 10 monclient: get_auth_request con 0x7f368c06ded0 auth_method 0 2024-03-25T19:44:08.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.189+0000 7f369affd640 10 client.4336.objecter ms_handle_connect 0x7f368c06ded0 2024-03-25T19:44:08.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.192+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f36a4054a90 con 0x7f36ac1430c0 2024-03-25T19:44:08.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.192+0000 7f369affd640 10 client.4336.objecter ms_dispatch 0x7f36ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:08.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.192+0000 7f369affd640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:08.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.192+0000 7f369affd640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:08.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.192+0000 7f369affd640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T19:44:08.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.193+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36a40503d0 con 0x7f36ac1430c0 2024-03-25T19:44:08.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.193+0000 7f369affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:08.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.193+0000 7f369affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:08.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.444+0000 7f36b0863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:08.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.444+0000 7f36b0863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:08.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.444+0000 7f36b0863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:08.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.444+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36ac069f40 con 0x7f36ac1430c0 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.643+0000 7f369affd640 1 -- 172.21.15.130:0/1169758254 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f36a40524d0 con 0x7f36ac1430c0 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.643+0000 7f369affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.643+0000 7f369affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:08.643 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:08.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 10 librados: watch_flush enter 2024-03-25T19:44:08.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 10 librados: watch_flush exit 2024-03-25T19:44:08.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T19:44:08.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 10 client.4336.objecter successfully canceled tick 2024-03-25T19:44:08.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 >> v1:172.21.15.163:6800/1217380380 conn(0x7f368c06ded0 legacy=0x7f368c070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 10 monclient: shutdown 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 >> v1:172.21.15.163:6789/0 conn(0x7f36ac1430c0 legacy=0x7f36ac133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 shutdown_connections 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.645+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 >> 172.21.15.130:0/1169758254 conn(0x7f36ac0b7ea0 msgr2=0x7f36ac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.646+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 shutdown_connections 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.646+0000 7f36b0863640 1 -- 172.21.15.130:0/1169758254 wait complete. 2024-03-25T19:44:08.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:08.646+0000 7f36b0863640 1 librados: shutdown 2024-03-25T19:44:08.657 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:15.463 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.577+0000 7fe4bf32b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.577+0000 7fe4bf32b640 10 monclient: build_initial_monmap 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.577+0000 7fe4bf32b640 10 monclient: monmap: 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:15.578466+0000 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:15.578466+0000 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:15.577 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:15.578 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.578+0000 7fe4bf32b640 1 Processor -- start 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.578+0000 7fe4bf32b640 1 -- start start 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.578+0000 7fe4bf32b640 10 monclient: init 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b81430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:15.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8142520 con 0x7fe4b81430c0 2024-03-25T19:44:15.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b28a0 con 0x7fe4b813f710 2024-03-25T19:44:15.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b80b2a30 con 0x7fe4b80a98f0 2024-03-25T19:44:15.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:15.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.579+0000 7fe4bf32b640 10 monclient(hunting): authenticate will time out at 5131.725342s 2024-03-25T19:44:15.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.586+0000 7fe4bc8c6640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe4b813f710 0x7fe4b81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:36120/0 (socket says 172.21.15.130:36120) 2024-03-25T19:44:15.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.586+0000 7fe4bc8c6640 1 -- 172.21.15.130:0/2477163770 learned_addr learned my addr 172.21.15.130:0/2477163770 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034151579 0 0) 0x7fe4b80b2a30 con 0x7fe4b80a98f0 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 10 monclient(hunting): my global_id is 4337 2024-03-25T19:44:15.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.602+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4b80a98f0 2024-03-25T19:44:15.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1342981797 0 0) 0x7fe4b80b28a0 con 0x7fe4b813f710 2024-03-25T19:44:15.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:15.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:15.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:15.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 10 monclient(hunting): my global_id is 4339 2024-03-25T19:44:15.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.604+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b80b2a30 con 0x7fe4b813f710 2024-03-25T19:44:15.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.614+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1289981507 0 0) 0x7fe4b80b2a30 con 0x7fe4b813f710 2024-03-25T19:44:15.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.614+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 >> v1:172.21.15.130:6790/0 conn(0x7fe4b80a98f0 legacy=0x7fe4b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.614+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 >> v1:172.21.15.130:6789/0 conn(0x7fe4b81430c0 legacy=0x7fe4b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 1 monclient: found mon.noname-c 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80b44c0 con 0x7fe4b813f710 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: _finish_auth 0 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe4a8001670 con 0x7fe4b813f710 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4affff640 10 monclient: dump: 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.615 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4bf32b640 5 monclient: authenticate success, global_id 4339 2024-03-25T19:44:15.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.615+0000 7fe4bf32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:15.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4a80017d0 con 0x7fe4b813f710 2024-03-25T19:44:15.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 1 -- 172.21.15.130:0/2477163770 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe4a8001ba0 con 0x7fe4b813f710 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4affff640 10 monclient: dump: 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:15.630 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:15.631 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:15.631 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4bf32b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:15.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.630+0000 7fe4bf32b640 4 set_mon_vals no callback set 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 10 monclient: shutdown 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2477163770 >> v1:172.21.15.163:6789/0 conn(0x7fe4b813f710 legacy=0x7fe4b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2477163770 shutdown_connections 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2477163770 >> 172.21.15.130:0/2477163770 conn(0x7fe4b80b7ea0 msgr2=0x7fe4b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2477163770 shutdown_connections 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.631+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2477163770 wait complete. 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 10 monclient: build_initial_monmap 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 10 monclient: monmap: 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:15.633 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 librados: starting msgr at 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 librados: starting objecter 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 Processor -- start 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 -- start start 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 librados: setting wanted keys 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 1 librados: calling monclient init 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.632+0000 7fe4bf32b640 10 monclient: init 2024-03-25T19:44:15.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-a con 0x7fe4b80a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-c con 0x7fe4b813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient(hunting): picked mon.noname-b con 0x7fe4b81430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b81d7750 con 0x7fe4b80a98f0 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b8136750 con 0x7fe4b81430c0 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4b81d85d0 con 0x7fe4b813f710 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.633+0000 7fe4bf32b640 10 monclient(hunting): authenticate will time out at 5131.779371s 2024-03-25T19:44:15.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.636+0000 7fe4bc8c6640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fe4b813f710 0x7fe4b8138300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:41820/0 (socket says 172.21.15.130:41820) 2024-03-25T19:44:15.636 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.636+0000 7fe4bc8c6640 1 -- 172.21.15.130:0/2634175071 learned_addr learned my addr 172.21.15.130:0/2634175071 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:15.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.642+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b813f710 2024-03-25T19:44:15.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.642+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:15.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.643+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b80a98f0 2024-03-25T19:44:15.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.643+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:15.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.653+0000 7fe4adffb640 10 client.?.objecter ms_handle_connect 0x7fe4b81430c0 2024-03-25T19:44:15.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.653+0000 7fe4adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023148221 0 0) 0x7fe4b81d85d0 con 0x7fe4b813f710 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4340 2024-03-25T19:44:15.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.657+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b813f710 2024-03-25T19:44:15.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.669+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 794207652 0 0) 0x7fe49c004eb0 con 0x7fe4b813f710 2024-03-25T19:44:15.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b81d85d0 con 0x7fe4b813f710 2024-03-25T19:44:15.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe4a80047e0 con 0x7fe4b813f710 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 10 monclient(hunting): dump: 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:15.670 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.670+0000 7fe4adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161473302 0 0) 0x7fe4b81d7750 con 0x7fe4b80a98f0 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4341 2024-03-25T19:44:15.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.674+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe49c004eb0 con 0x7fe4b80a98f0 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 71666398 0 0) 0x7fe4b8136750 con 0x7fe4b81430c0 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 10 monclient(hunting): my global_id is 4342 2024-03-25T19:44:15.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.675+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4b81d7750 con 0x7fe4b81430c0 2024-03-25T19:44:15.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3920886757 0 0) 0x7fe4b81d7750 con 0x7fe4b81430c0 2024-03-25T19:44:15.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4b8136750 con 0x7fe4b81430c0 2024-03-25T19:44:15.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe4b00016e0 con 0x7fe4b81430c0 2024-03-25T19:44:15.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.684+0000 7fe4adffb640 10 monclient(hunting): dump: 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:15.684 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 236444943 0 0) 0x7fe4b81d85d0 con 0x7fe4b813f710 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 >> v1:172.21.15.163:6789/0 conn(0x7fe4b81430c0 legacy=0x7fe4b81d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 >> v1:172.21.15.130:6789/0 conn(0x7fe4b80a98f0 legacy=0x7fe4b81d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 1 monclient: found mon.c 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b80ab950 con 0x7fe4b813f710 2024-03-25T19:44:15.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 5 monclient: authenticate success, global_id 4340 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 10 monclient: _renew_subs 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b81d7a00 con 0x7fe4b813f710 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 10 monclient: _renew_subs 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b8138e60 con 0x7fe4b813f710 2024-03-25T19:44:15.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.691+0000 7fe4bf32b640 1 librados: init done 2024-03-25T19:44:15.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.692+0000 7fe4bf32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:15.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.692+0000 7fe4bf32b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:15.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.692+0000 7fe4bf32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:44:15.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.692+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b80627b0 con 0x7fe4b813f710 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe4a8004bf0 con 0x7fe4b813f710 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe4a8004f00 con 0x7fe4b813f710 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4adffb640 10 monclient: dump: 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:15.999 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.998+0000 7fe4af7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.999+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fe4a801aed0 con 0x7fe4b813f710 2024-03-25T19:44:16.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:15.999+0000 7fe4bd0c7640 10 monclient: get_auth_request con 0x7fe49c06df80 auth_method 0 2024-03-25T19:44:16.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.000+0000 7fe4adffb640 10 client.4340.objecter ms_handle_connect 0x7fe49c06df80 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fe4a80549a0 con 0x7fe4b813f710 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 10 client.4340.objecter ms_dispatch 0x7fe4b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe4a80502e0 con 0x7fe4b813f710 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:16.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.005+0000 7fe4adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:16.242 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.241+0000 7fe4bf32b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.242 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.241+0000 7fe4bf32b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.242 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.241+0000 7fe4bf32b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:44:16.242 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.241+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4b8074cc0 con 0x7fe4b813f710 2024-03-25T19:44:16.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.325+0000 7fe4adffb640 1 -- 172.21.15.130:0/2634175071 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fe4a80523e0 con 0x7fe4b813f710 2024-03-25T19:44:16.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.325+0000 7fe4adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:16.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.325+0000 7fe4adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:16.326 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:16.326 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 10 librados: watch_flush enter 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 10 librados: watch_flush exit 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 10 client.4340.objecter successfully canceled tick 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 >> v1:172.21.15.163:6800/1217380380 conn(0x7fe49c06df80 legacy=0x7fe49c070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 10 monclient: shutdown 2024-03-25T19:44:16.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 >> v1:172.21.15.130:6790/0 conn(0x7fe4b813f710 legacy=0x7fe4b8138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 shutdown_connections 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 >> 172.21.15.130:0/2634175071 conn(0x7fe4b80b7ea0 msgr2=0x7fe4b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 shutdown_connections 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 -- 172.21.15.130:0/2634175071 wait complete. 2024-03-25T19:44:16.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:16.327+0000 7fe4bf32b640 1 librados: shutdown 2024-03-25T19:44:16.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:19.752 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:04.685 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.765 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:04.699 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.791 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:04.725 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.826 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:04.727 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.843 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:04.777 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.844 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:04.745 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.856 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:04.789 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.872 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:04.774 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:19.880 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:04.813 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:23.146 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:23.263 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.263+0000 7f903160f640 10 monclient: get_monmap_and_config 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.263+0000 7f903160f640 10 monclient: build_initial_monmap 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.263+0000 7f903160f640 10 monclient: monmap: 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:23.265167+0000 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:23.265167+0000 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:23.264 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.263+0000 7f903160f640 1 Processor -- start 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 1 -- start start 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient: init 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient(hunting): picked mon.noname-c con 0x7f902c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient(hunting): picked mon.noname-a con 0x7f902c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient(hunting): picked mon.noname-b con 0x7f902c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:23.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c142520 con 0x7f902c13f710 2024-03-25T19:44:23.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c0b28a0 con 0x7f902c0a98f0 2024-03-25T19:44:23.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c0b2a30 con 0x7f902c1430c0 2024-03-25T19:44:23.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:23.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.264+0000 7f903160f640 10 monclient(hunting): authenticate will time out at 5139.411250s 2024-03-25T19:44:23.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.269+0000 7f902b7fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f902c1430c0 0x7f902c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:60696/0 (socket says 172.21.15.130:60696) 2024-03-25T19:44:23.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.269+0000 7f902b7fe640 1 -- 172.21.15.130:0/321372766 learned_addr learned my addr 172.21.15.130:0/321372766 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:23.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 949697203 0 0) 0x7f902c0b2a30 con 0x7f902c1430c0 2024-03-25T19:44:23.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 10 monclient(hunting): my global_id is 4343 2024-03-25T19:44:23.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.279+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9014003520 con 0x7f902c1430c0 2024-03-25T19:44:23.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820667828 0 0) 0x7f902c142520 con 0x7f902c13f710 2024-03-25T19:44:23.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 10 monclient(hunting): my global_id is 4344 2024-03-25T19:44:23.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.282+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c0b2a30 con 0x7f902c13f710 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268088046 0 0) 0x7f902c0b28a0 con 0x7f902c0a98f0 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T19:44:23.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.286+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f902c142520 con 0x7f902c0a98f0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2603294630 0 0) 0x7f9014003520 con 0x7f902c1430c0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 >> v1:172.21.15.163:6789/0 conn(0x7f902c0a98f0 legacy=0x7f902c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 >> v1:172.21.15.130:6789/0 conn(0x7f902c13f710 legacy=0x7f902c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 monclient: found mon.noname-b 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c0b44c0 con 0x7f902c1430c0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:23.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9024001a10 con 0x7f902c1430c0 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f9029ffb640 10 monclient: dump: 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:23.296 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f903160f640 5 monclient: authenticate success, global_id 4343 2024-03-25T19:44:23.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.294+0000 7f903160f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9024001b70 con 0x7f902c1430c0 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 1 -- 172.21.15.130:0/321372766 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9024001f80 con 0x7f902c1430c0 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.305+0000 7f9029ffb640 10 monclient: dump: 2024-03-25T19:44:23.306 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:23.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 4 set_mon_vals no callback set 2024-03-25T19:44:23.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 10 monclient: shutdown 2024-03-25T19:44:23.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:23.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 1 -- 172.21.15.130:0/321372766 >> v1:172.21.15.130:6790/0 conn(0x7f902c1430c0 legacy=0x7f902c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 1 -- 172.21.15.130:0/321372766 shutdown_connections 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 1 -- 172.21.15.130:0/321372766 >> 172.21.15.130:0/321372766 conn(0x7f902c0b7ea0 msgr2=0x7f902c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 1 -- 172.21.15.130:0/321372766 shutdown_connections 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.306+0000 7f903160f640 1 -- 172.21.15.130:0/321372766 wait complete. 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.307+0000 7f903160f640 10 monclient: build_initial_monmap 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.307+0000 7f903160f640 10 monclient: monmap: 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:23.310 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.307+0000 7f903160f640 1 librados: starting msgr at 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.307+0000 7f903160f640 1 librados: starting objecter 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 Processor -- start 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 -- start start 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 librados: setting wanted keys 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 librados: calling monclient init 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient: init 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient(hunting): picked mon.noname-c con 0x7f902c0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient(hunting): picked mon.noname-b con 0x7f902c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient(hunting): picked mon.noname-a con 0x7f902c1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c137ab0 con 0x7f902c1430c0 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c138bd0 con 0x7f902c13f710 2024-03-25T19:44:23.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f902c138e40 con 0x7f902c0a98f0 2024-03-25T19:44:23.312 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:23.312 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.308+0000 7f903160f640 10 monclient(hunting): authenticate will time out at 5139.455593s 2024-03-25T19:44:23.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.315+0000 7f902b7fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f902c1430c0 0x7f902c1e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:44346/0 (socket says 172.21.15.130:44346) 2024-03-25T19:44:23.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.315+0000 7f902b7fe640 1 -- 172.21.15.130:0/1904664197 learned_addr learned my addr 172.21.15.130:0/1904664197 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:23.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.316+0000 7f90137fe640 10 client.?.objecter ms_handle_connect 0x7f902c1430c0 2024-03-25T19:44:23.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.317+0000 7f90137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702106286 0 0) 0x7f902c137ab0 con 0x7f902c1430c0 2024-03-25T19:44:23.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T19:44:23.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.324+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9018004eb0 con 0x7f902c1430c0 2024-03-25T19:44:23.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.327+0000 7f90137fe640 10 client.?.objecter ms_handle_connect 0x7f902c13f710 2024-03-25T19:44:23.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.327+0000 7f90137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2396932149 0 0) 0x7f9018004eb0 con 0x7f902c1430c0 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f902c137ab0 con 0x7f902c1430c0 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9024001ec0 con 0x7f902c1430c0 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 10 monclient(hunting): dump: 2024-03-25T19:44:23.337 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.336+0000 7f90137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:23.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 client.?.objecter ms_handle_connect 0x7f902c0a98f0 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2247247641 0 0) 0x7f902c138bd0 con 0x7f902c13f710 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 10 monclient(hunting): my global_id is 4348 2024-03-25T19:44:23.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.341+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9018004eb0 con 0x7f902c13f710 2024-03-25T19:44:23.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243115495 0 0) 0x7f9018004eb0 con 0x7f902c13f710 2024-03-25T19:44:23.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f902c138bd0 con 0x7f902c13f710 2024-03-25T19:44:23.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9020001930 con 0x7f902c13f710 2024-03-25T19:44:23.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.360+0000 7f90137fe640 10 monclient(hunting): dump: 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:23.362 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458149844 0 0) 0x7f902c138e40 con 0x7f902c0a98f0 2024-03-25T19:44:23.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:23.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:23.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:23.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 10 monclient(hunting): my global_id is 4346 2024-03-25T19:44:23.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.367+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9018004eb0 con 0x7f902c0a98f0 2024-03-25T19:44:23.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.368+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1970483031 0 0) 0x7f902c137ab0 con 0x7f902c1430c0 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 >> v1:172.21.15.130:6790/0 conn(0x7f902c0a98f0 legacy=0x7f902c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 >> v1:172.21.15.163:6789/0 conn(0x7f902c13f710 legacy=0x7f902c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 1 monclient: found mon.a 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f902c139120 con 0x7f902c1430c0 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f90137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 5 monclient: authenticate success, global_id 4347 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 10 monclient: _renew_subs 2024-03-25T19:44:23.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f902c139400 con 0x7f902c1430c0 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 10 monclient: _renew_subs 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f902c0b7880 con 0x7f902c1430c0 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.369+0000 7f903160f640 1 librados: init done 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.370+0000 7f903160f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:23.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.370+0000 7f903160f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:23.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.370+0000 7f903160f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:23.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.370+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f902c137810 con 0x7f902c1430c0 2024-03-25T19:44:23.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9024004e10 con 0x7f902c1430c0 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9024005120 con 0x7f902c1430c0 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90137fe640 10 monclient: dump: 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:23.689 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.688+0000 7f90297fa640 4 set_mon_vals no callback set 2024-03-25T19:44:23.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.689+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f902401b0f0 con 0x7f902c1430c0 2024-03-25T19:44:23.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.689+0000 7f902affd640 10 monclient: get_auth_request con 0x7f901806df70 auth_method 0 2024-03-25T19:44:23.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.690+0000 7f90137fe640 10 client.4347.objecter ms_handle_connect 0x7f901806df70 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f9024054bd0 con 0x7f902c1430c0 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 10 client.4347.objecter ms_dispatch 0x7f902c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f90240506e0 con 0x7f902c1430c0 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:23.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.694+0000 7f90137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:23.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.943+0000 7f903160f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.943+0000 7f903160f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.943+0000 7f903160f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:23.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.943+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f902c137c60 con 0x7f902c1430c0 2024-03-25T19:44:23.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.977+0000 7f90137fe640 1 -- 172.21.15.130:0/1904664197 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f902c137c60 con 0x7f902c1430c0 2024-03-25T19:44:23.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.977+0000 7f90137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:23.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.977+0000 7f90137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:23.978 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:23.978 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 10 librados: watch_flush enter 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 10 librados: watch_flush exit 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 10 client.4347.objecter successfully canceled tick 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 >> v1:172.21.15.163:6800/1217380380 conn(0x7f901806df70 legacy=0x7f9018070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 10 monclient: shutdown 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 >> v1:172.21.15.130:6789/0 conn(0x7f902c1430c0 legacy=0x7f902c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 shutdown_connections 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 >> 172.21.15.130:0/1904664197 conn(0x7f902c0b7ea0 msgr2=0x7f902c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:23.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 shutdown_connections 2024-03-25T19:44:23.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 -- 172.21.15.130:0/1904664197 wait complete. 2024-03-25T19:44:23.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:23.980+0000 7f903160f640 1 librados: shutdown 2024-03-25T19:44:23.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:30.800 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:30.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.918+0000 7f7f02a86640 10 monclient: get_monmap_and_config 2024-03-25T19:44:30.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.918+0000 7f7f02a86640 10 monclient: build_initial_monmap 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.918+0000 7f7f02a86640 10 monclient: monmap: 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:30.919069+0000 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:30.919069+0000 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:30.918 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:30.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.919+0000 7f7f02a86640 1 Processor -- start 2024-03-25T19:44:30.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.919+0000 7f7f02a86640 1 -- start start 2024-03-25T19:44:30.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.919+0000 7f7f02a86640 10 monclient: init 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-a con 0x7f7efc0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-c con 0x7f7efc0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-b con 0x7f7efc14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0af940 con 0x7f7efc0b4d60 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0b0a70 con 0x7f7efc0b1730 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc0b0d50 con 0x7f7efc14be90 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:30.920 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.920+0000 7f7f02a86640 10 monclient(hunting): authenticate will time out at 5147.065935s 2024-03-25T19:44:30.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.925+0000 7f7f01023640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f7efc14be90 0x7f7efc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51474/0 (socket says 172.21.15.130:51474) 2024-03-25T19:44:30.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.925+0000 7f7f01023640 1 -- 172.21.15.130:0/3704496880 learned_addr learned my addr 172.21.15.130:0/3704496880 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553338407 0 0) 0x7f7efc0b0d50 con 0x7f7efc14be90 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 10 monclient(hunting): my global_id is 4349 2024-03-25T19:44:30.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.935+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7eec003520 con 0x7f7efc14be90 2024-03-25T19:44:30.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196417644 0 0) 0x7f7efc0b0a70 con 0x7f7efc0b1730 2024-03-25T19:44:30.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 10 monclient(hunting): my global_id is 4351 2024-03-25T19:44:30.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.954+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7efc0b0d50 con 0x7f7efc0b1730 2024-03-25T19:44:30.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258697431 0 0) 0x7f7eec003520 con 0x7f7efc14be90 2024-03-25T19:44:30.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 >> v1:172.21.15.163:6789/0 conn(0x7f7efc0b1730 legacy=0x7f7efc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 >> v1:172.21.15.130:6789/0 conn(0x7f7efc0b4d60 legacy=0x7f7efc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 monclient: found mon.noname-b 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0b1030 con 0x7f7efc14be90 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7ef4001a10 con 0x7f7efc14be90 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7efb7fe640 10 monclient: dump: 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:30.959 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7f02a86640 5 monclient: authenticate success, global_id 4349 2024-03-25T19:44:30.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.958+0000 7f7f02a86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ef4001b70 con 0x7f7efc14be90 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 1 -- 172.21.15.130:0/3704496880 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7ef4001f80 con 0x7f7efc14be90 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7efb7fe640 10 monclient: dump: 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:30.974 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7f02a86640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:30.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.974+0000 7f7f02a86640 4 set_mon_vals no callback set 2024-03-25T19:44:30.977 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 10 monclient: shutdown 2024-03-25T19:44:30.977 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 1 -- 172.21.15.130:0/3704496880 >> v1:172.21.15.130:6790/0 conn(0x7f7efc14be90 legacy=0x7f7efc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 1 -- 172.21.15.130:0/3704496880 shutdown_connections 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 1 -- 172.21.15.130:0/3704496880 >> 172.21.15.130:0/3704496880 conn(0x7f7efc0617a0 msgr2=0x7f7efc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 1 -- 172.21.15.130:0/3704496880 shutdown_connections 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.975+0000 7f7f02a86640 1 -- 172.21.15.130:0/3704496880 wait complete. 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 10 monclient: build_initial_monmap 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 10 monclient: monmap: 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:30.978 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 librados: starting msgr at 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 librados: starting objecter 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 Processor -- start 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 -- start start 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 librados: setting wanted keys 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 1 librados: calling monclient init 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.976+0000 7f7f02a86640 10 monclient: init 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-b con 0x7f7efc0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:30.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-c con 0x7f7efc0b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient(hunting): picked mon.noname-a con 0x7f7efc14be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc137380 con 0x7f7efc14be90 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc140760 con 0x7f7efc0b1730 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc140380 con 0x7f7efc0b4d60 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.977+0000 7f7f02a86640 10 monclient(hunting): authenticate will time out at 5147.123174s 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.981+0000 7f7efbfff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f7efc0b4d60 0x7f7efc1dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51538/0 (socket says 172.21.15.130:51538) 2024-03-25T19:44:30.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.981+0000 7f7efbfff640 1 -- 172.21.15.130:0/630625941 learned_addr learned my addr 172.21.15.130:0/630625941 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:30.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.981+0000 7f7ef97fa640 10 client.?.objecter ms_handle_connect 0x7f7efc0b4d60 2024-03-25T19:44:30.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.981+0000 7f7ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:30.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.987+0000 7f7ef97fa640 10 client.?.objecter ms_handle_connect 0x7f7efc14be90 2024-03-25T19:44:30.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.987+0000 7f7ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050864028 0 0) 0x7f7efc140380 con 0x7f7efc0b4d60 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 10 monclient(hunting): my global_id is 4352 2024-03-25T19:44:30.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.991+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee0004eb0 con 0x7f7efc0b4d60 2024-03-25T19:44:30.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.993+0000 7f7ef97fa640 10 client.?.objecter ms_handle_connect 0x7f7efc0b1730 2024-03-25T19:44:30.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:30.993+0000 7f7ef97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520582431 0 0) 0x7f7efc137380 con 0x7f7efc14be90 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 10 monclient(hunting): my global_id is 4353 2024-03-25T19:44:31.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.002+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7efc140380 con 0x7f7efc14be90 2024-03-25T19:44:31.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.025+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3889461298 0 0) 0x7f7efc140380 con 0x7f7efc14be90 2024-03-25T19:44:31.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7efc137380 con 0x7f7efc14be90 2024-03-25T19:44:31.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7ef4001e80 con 0x7f7efc14be90 2024-03-25T19:44:31.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 10 monclient(hunting): dump: 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:31.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.026+0000 7f7ef97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36494810 0 0) 0x7f7efc137380 con 0x7f7efc14be90 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 >> v1:172.21.15.130:6790/0 conn(0x7f7efc0b4d60 legacy=0x7f7efc1dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 >> v1:172.21.15.163:6789/0 conn(0x7f7efc0b1730 legacy=0x7f7efc134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 1 monclient: found mon.a 2024-03-25T19:44:31.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc140510 con 0x7f7efc14be90 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 10 monclient: _finish_auth 0 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7ef97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 5 monclient: authenticate success, global_id 4353 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 10 monclient: _renew_subs 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc135650 con 0x7f7efc14be90 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 10 monclient: _renew_subs 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:31.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc144da0 con 0x7f7efc14be90 2024-03-25T19:44:31.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.095+0000 7f7f02a86640 1 librados: init done 2024-03-25T19:44:31.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.096+0000 7f7f02a86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.097+0000 7f7f02a86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.097+0000 7f7f02a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:31.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.097+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc0627b0 con 0x7f7efc14be90 2024-03-25T19:44:31.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ef40015c0 con 0x7f7efc14be90 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7ef4004e10 con 0x7f7efc14be90 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7ef97fa640 10 monclient: dump: 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:31.388 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.387+0000 7f7efaffd640 4 set_mon_vals no callback set 2024-03-25T19:44:31.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.388+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f7ef401ade0 con 0x7f7efc14be90 2024-03-25T19:44:31.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.389+0000 7f7f00822640 10 monclient: get_auth_request con 0x7f7ee006dd90 auth_method 0 2024-03-25T19:44:31.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.389+0000 7f7ef97fa640 10 client.4353.objecter ms_handle_connect 0x7f7ee006dd90 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f7ef4054930 con 0x7f7efc14be90 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 10 client.4353.objecter ms_dispatch 0x7f7efc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ef4050270 con 0x7f7efc14be90 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:31.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.394+0000 7f7ef97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:31.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.646+0000 7f7f02a86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.646+0000 7f7f02a86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.647+0000 7f7f02a86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:44:31.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.647+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7efc069f40 con 0x7f7efc14be90 2024-03-25T19:44:31.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.686+0000 7f7ef97fa640 1 -- 172.21.15.130:0/630625941 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f7ef4052370 con 0x7f7efc14be90 2024-03-25T19:44:31.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.686+0000 7f7ef97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:31.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.686+0000 7f7ef97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:31.686 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:31.686 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:31.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 10 librados: watch_flush enter 2024-03-25T19:44:31.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 10 librados: watch_flush exit 2024-03-25T19:44:31.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T19:44:31.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 10 client.4353.objecter successfully canceled tick 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 >> v1:172.21.15.163:6800/1217380380 conn(0x7f7ee006dd90 legacy=0x7f7ee0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 10 monclient: shutdown 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 >> v1:172.21.15.130:6789/0 conn(0x7f7efc14be90 legacy=0x7f7efc1db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 shutdown_connections 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 >> 172.21.15.130:0/630625941 conn(0x7f7efc0617a0 msgr2=0x7f7efc0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 shutdown_connections 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 -- 172.21.15.130:0/630625941 wait complete. 2024-03-25T19:44:31.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:31.689+0000 7f7f02a86640 1 librados: shutdown 2024-03-25T19:44:31.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:38.508 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:38.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.620+0000 7f4f9b6e6640 10 monclient: get_monmap_and_config 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.620+0000 7f4f9b6e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.620+0000 7f4f9b6e6640 10 monclient: monmap: 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:38.621303+0000 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:38.621303+0000 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:38.620 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.621+0000 7f4f9b6e6640 1 Processor -- start 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.621+0000 7f4f9b6e6640 1 -- start start 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.621+0000 7f4f9b6e6640 10 monclient: init 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f9413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f941430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f94142520 con 0x7f4f941430c0 2024-03-25T19:44:38.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940b28a0 con 0x7f4f940a98f0 2024-03-25T19:44:38.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f940b2a30 con 0x7f4f9413f710 2024-03-25T19:44:38.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:38.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.622+0000 7f4f9b6e6640 10 monclient(hunting): authenticate will time out at 5154.768158s 2024-03-25T19:44:38.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.643+0000 7f4f99482640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4f940a98f0 0x7f4f940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:40400/0 (socket says 172.21.15.130:40400) 2024-03-25T19:44:38.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.643+0000 7f4f99482640 1 -- 172.21.15.130:0/2953230198 learned_addr learned my addr 172.21.15.130:0/2953230198 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827822455 0 0) 0x7f4f940b2a30 con 0x7f4f9413f710 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 10 monclient(hunting): my global_id is 4355 2024-03-25T19:44:38.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.661+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f80003520 con 0x7f4f9413f710 2024-03-25T19:44:38.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26282984 0 0) 0x7f4f94142520 con 0x7f4f941430c0 2024-03-25T19:44:38.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 10 monclient(hunting): my global_id is 4356 2024-03-25T19:44:38.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.663+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f940b2a30 con 0x7f4f941430c0 2024-03-25T19:44:38.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143691129 0 0) 0x7f4f940b28a0 con 0x7f4f940a98f0 2024-03-25T19:44:38.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 10 monclient(hunting): my global_id is 4357 2024-03-25T19:44:38.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.666+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f94142520 con 0x7f4f940a98f0 2024-03-25T19:44:38.679 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.679+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2351215150 0 0) 0x7f4f80003520 con 0x7f4f9413f710 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 >> v1:172.21.15.163:6789/0 conn(0x7f4f940a98f0 legacy=0x7f4f940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 >> v1:172.21.15.130:6789/0 conn(0x7f4f941430c0 legacy=0x7f4f9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 1 monclient: found mon.noname-b 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f940b44c0 con 0x7f4f9413f710 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: _finish_auth 0 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4f84001690 con 0x7f4f9413f710 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f9b6e6640 5 monclient: authenticate success, global_id 4355 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f9b6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:38.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f8bfff640 10 monclient: dump: 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:38.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.680+0000 7f4f9b6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f840017f0 con 0x7f4f9413f710 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 1 -- 172.21.15.130:0/2953230198 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4f84001c00 con 0x7f4f9413f710 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 4 set_mon_vals no callback set 2024-03-25T19:44:38.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f8bfff640 10 monclient: dump: 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:38.718 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:38.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 10 monclient: shutdown 2024-03-25T19:44:38.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:38.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/2953230198 >> v1:172.21.15.130:6790/0 conn(0x7f4f9413f710 legacy=0x7f4f941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/2953230198 shutdown_connections 2024-03-25T19:44:38.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/2953230198 >> 172.21.15.130:0/2953230198 conn(0x7f4f940b7ea0 msgr2=0x7f4f940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.718+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/2953230198 shutdown_connections 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.719+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/2953230198 wait complete. 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.719+0000 7f4f9b6e6640 10 monclient: build_initial_monmap 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.719+0000 7f4f9b6e6640 10 monclient: monmap: 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.719+0000 7f4f9b6e6640 1 librados: starting msgr at 2024-03-25T19:44:38.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.719+0000 7f4f9b6e6640 1 librados: starting objecter 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 1 Processor -- start 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 1 -- start start 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 1 librados: setting wanted keys 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 1 librados: calling monclient init 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 10 monclient: init 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.720+0000 7f4f9b6e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-c con 0x7f4f9413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f4f941430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f9413be70 con 0x7f4f941430c0 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f94131660 con 0x7f4f940a98f0 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f941317f0 con 0x7f4f9413f710 2024-03-25T19:44:38.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:38.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.721+0000 7f4f9b6e6640 10 monclient(hunting): authenticate will time out at 5154.866697s 2024-03-25T19:44:38.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.730+0000 7f4f99482640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4f940a98f0 0x7f4f94130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:40454/0 (socket says 172.21.15.130:40454) 2024-03-25T19:44:38.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.730+0000 7f4f99482640 1 -- 172.21.15.130:0/4266308095 learned_addr learned my addr 172.21.15.130:0/4266308095 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:38.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.734+0000 7f4f89ffb640 10 client.?.objecter ms_handle_connect 0x7f4f940a98f0 2024-03-25T19:44:38.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.734+0000 7f4f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:38.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.738+0000 7f4f89ffb640 10 client.?.objecter ms_handle_connect 0x7f4f9413f710 2024-03-25T19:44:38.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.738+0000 7f4f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:38.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.744+0000 7f4f89ffb640 10 client.?.objecter ms_handle_connect 0x7f4f941430c0 2024-03-25T19:44:38.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.744+0000 7f4f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172300275 0 0) 0x7f4f94131660 con 0x7f4f940a98f0 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78004eb0 con 0x7f4f940a98f0 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308492754 0 0) 0x7f4f941317f0 con 0x7f4f9413f710 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 10 monclient(hunting): my global_id is 4358 2024-03-25T19:44:38.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.751+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f94131660 con 0x7f4f9413f710 2024-03-25T19:44:38.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065837584 0 0) 0x7f4f9413be70 con 0x7f4f941430c0 2024-03-25T19:44:38.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:38.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:38.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:38.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 10 monclient(hunting): my global_id is 4359 2024-03-25T19:44:38.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.757+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f941317f0 con 0x7f4f941430c0 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4210500407 0 0) 0x7f4f78004eb0 con 0x7f4f940a98f0 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f9413be70 con 0x7f4f940a98f0 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4f90001910 con 0x7f4f940a98f0 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 10 monclient(hunting): dump: 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.771 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:38.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.771+0000 7f4f89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 268508726 0 0) 0x7f4f9413be70 con 0x7f4f940a98f0 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 >> v1:172.21.15.130:6790/0 conn(0x7f4f9413f710 legacy=0x7f4f941d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 >> v1:172.21.15.130:6789/0 conn(0x7f4f941430c0 legacy=0x7f4f941e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 1 monclient: found mon.b 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:38.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f94131980 con 0x7f4f940a98f0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f9b6e6640 5 monclient: authenticate success, global_id 4360 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f9b6e6640 10 monclient: _renew_subs 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f9b6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.786+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f94135c60 con 0x7f4f940a98f0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.787+0000 7f4f9b6e6640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.787+0000 7f4f9b6e6640 10 monclient: _renew_subs 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.787+0000 7f4f9b6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.787+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f940b7770 con 0x7f4f940a98f0 2024-03-25T19:44:38.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.787+0000 7f4f9b6e6640 1 librados: init done 2024-03-25T19:44:38.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.788+0000 7f4f9b6e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:38.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.788+0000 7f4f9b6e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:38.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.788+0000 7f4f9b6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:38.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:38.788+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f941371d0 con 0x7f4f940a98f0 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f900047e0 con 0x7f4f940a98f0 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4f90004bf0 con 0x7f4f940a98f0 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:39.073 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f89ffb640 10 monclient: dump: 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:39.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.073+0000 7f4f8b7fe640 4 set_mon_vals no callback set 2024-03-25T19:44:39.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.074+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f4f90004e10 con 0x7f4f940a98f0 2024-03-25T19:44:39.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.075+0000 7f4f98c81640 10 monclient: get_auth_request con 0x7f4f7806de70 auth_method 0 2024-03-25T19:44:39.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.075+0000 7f4f89ffb640 10 client.4360.objecter ms_handle_connect 0x7f4f7806de70 2024-03-25T19:44:39.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.079+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f4f900548f0 con 0x7f4f940a98f0 2024-03-25T19:44:39.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.079+0000 7f4f89ffb640 10 client.4360.objecter ms_dispatch 0x7f4f94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:39.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.079+0000 7f4f89ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:39.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.079+0000 7f4f89ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:39.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.079+0000 7f4f89ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T19:44:39.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.081+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4f90050400 con 0x7f4f940a98f0 2024-03-25T19:44:39.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.081+0000 7f4f89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:39.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.081+0000 7f4f89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:39.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.299+0000 7f4f9b6e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.300+0000 7f4f9b6e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.300+0000 7f4f9b6e6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:39.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.300+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f940627b0 con 0x7f4f940a98f0 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.331+0000 7f4f89ffb640 1 -- 172.21.15.130:0/4266308095 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f4f90052330 con 0x7f4f940a98f0 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.331+0000 7f4f89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.331+0000 7f4f89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:39.331 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:39.333 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.333+0000 7f4f9b6e6640 10 librados: watch_flush enter 2024-03-25T19:44:39.333 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.333+0000 7f4f9b6e6640 10 librados: watch_flush exit 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.333+0000 7f4f9b6e6640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.333+0000 7f4f9b6e6640 10 client.4360.objecter successfully canceled tick 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 >> v1:172.21.15.163:6800/1217380380 conn(0x7f4f7806de70 legacy=0x7f4f78070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 10 monclient: shutdown 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 >> v1:172.21.15.163:6789/0 conn(0x7f4f940a98f0 legacy=0x7f4f94130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 shutdown_connections 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 >> 172.21.15.130:0/4266308095 conn(0x7f4f940b7ea0 msgr2=0x7f4f94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 shutdown_connections 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 -- 172.21.15.130:0/4266308095 wait complete. 2024-03-25T19:44:39.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:39.334+0000 7f4f9b6e6640 1 librados: shutdown 2024-03-25T19:44:39.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:39.768 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:24.702 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.827 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:24.760 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.876 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:24.777 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.877 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:24.810 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.885 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:24.786 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.888 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:24.821 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.901 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:24.834 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.925 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:24.826 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:39.934 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:24.867 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:46.153 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:46.270 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.271+0000 7fe2fcc7b640 10 monclient: get_monmap_and_config 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.271+0000 7fe2fcc7b640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.271+0000 7fe2fcc7b640 10 monclient: monmap: 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:46.272356+0000 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:46.272356+0000 2024-03-25T19:44:46.271 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.272 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.272 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:46.272 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:46.272 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:46.272 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 1 Processor -- start 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 1 -- start start 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient: init 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f80b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f80b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80af940 con 0x7fe2f80b1730 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80b0a70 con 0x7fe2f814be90 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80b0d50 con 0x7fe2f80b4d60 2024-03-25T19:44:46.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.272+0000 7fe2fcc7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.273+0000 7fe2fcc7b640 10 monclient(hunting): authenticate will time out at 5162.418618s 2024-03-25T19:44:46.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.285+0000 7fe2f5d9b640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fe2f80b1730 0x7fe2f80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:57898/0 (socket says 172.21.15.130:57898) 2024-03-25T19:44:46.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.285+0000 7fe2f5d9b640 1 -- 172.21.15.130:0/3909536477 learned_addr learned my addr 172.21.15.130:0/3909536477 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98289629 0 0) 0x7fe2f80af940 con 0x7fe2f80b1730 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 10 monclient(hunting): my global_id is 4362 2024-03-25T19:44:46.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.308+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2e0003520 con 0x7fe2f80b1730 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462000903 0 0) 0x7fe2f80b0a70 con 0x7fe2f814be90 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 10 monclient(hunting): my global_id is 4363 2024-03-25T19:44:46.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.314+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f80af940 con 0x7fe2f814be90 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1717032246 0 0) 0x7fe2f80af940 con 0x7fe2f814be90 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 >> v1:172.21.15.130:6790/0 conn(0x7fe2f80b4d60 legacy=0x7fe2f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 >> v1:172.21.15.130:6789/0 conn(0x7fe2f80b1730 legacy=0x7fe2f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 1 monclient: found mon.noname-c 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f80b1030 con 0x7fe2f814be90 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2fcc7b640 5 monclient: authenticate success, global_id 4363 2024-03-25T19:44:46.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.322+0000 7fe2fcc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.325+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe2f00019b0 con 0x7fe2f814be90 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.325+0000 7fe2f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.325+0000 7fe2f559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.325+0000 7fe2f559a640 10 monclient: dump: 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:46.325 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:46.326 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:46.326 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.325+0000 7fe2fcc7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2f0001c40 con 0x7fe2f814be90 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 1 -- 172.21.15.130:0/3909536477 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe2f0001f50 con 0x7fe2f814be90 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2fcc7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2fcc7b640 4 set_mon_vals no callback set 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.338+0000 7fe2f559a640 10 monclient: dump: 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.339 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 10 monclient: shutdown 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/3909536477 >> v1:172.21.15.163:6789/0 conn(0x7fe2f814be90 legacy=0x7fe2f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/3909536477 shutdown_connections 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/3909536477 >> 172.21.15.130:0/3909536477 conn(0x7fe2f80617a0 msgr2=0x7fe2f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/3909536477 shutdown_connections 2024-03-25T19:44:46.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.339+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/3909536477 wait complete. 2024-03-25T19:44:46.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.340+0000 7fe2fcc7b640 10 monclient: build_initial_monmap 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.340+0000 7fe2fcc7b640 10 monclient: monmap: 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 librados: starting msgr at 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 librados: starting objecter 2024-03-25T19:44:46.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 Processor -- start 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 -- start start 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 librados: setting wanted keys 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 librados: calling monclient init 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient: init 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe2f80b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe2f80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe2f814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80a9df0 con 0x7fe2f80b4d60 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.341+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f813bf00 con 0x7fe2f80b1730 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.342+0000 7fe2fcc7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2f80aab70 con 0x7fe2f814be90 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.342+0000 7fe2fcc7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:46.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.342+0000 7fe2fcc7b640 10 monclient(hunting): authenticate will time out at 5162.487629s 2024-03-25T19:44:46.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.354+0000 7fe2f659c640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fe2f80b1730 0x7fe2f80a96e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:45128/0 (socket says 172.21.15.130:45128) 2024-03-25T19:44:46.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.354+0000 7fe2f5d9b640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fe2f80b4d60 0x7fe2f81d7d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:57942/0 (socket says 172.21.15.130:57942) 2024-03-25T19:44:46.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.354+0000 7fe2f659c640 1 -- 172.21.15.130:0/2726351287 learned_addr learned my addr 172.21.15.130:0/2726351287 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:46.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.358+0000 7fe2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe2f80b1730 2024-03-25T19:44:46.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.358+0000 7fe2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.359+0000 7fe2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe2f80b4d60 2024-03-25T19:44:46.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.359+0000 7fe2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.359+0000 7fe2e6ffd640 10 client.?.objecter ms_handle_connect 0x7fe2f814be90 2024-03-25T19:44:46.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.359+0000 7fe2e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:46.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446955901 0 0) 0x7fe2f80a9df0 con 0x7fe2f80b4d60 2024-03-25T19:44:46.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 10 monclient(hunting): my global_id is 4365 2024-03-25T19:44:46.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.369+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d8004eb0 con 0x7fe2f80b4d60 2024-03-25T19:44:46.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910818400 0 0) 0x7fe2f813bf00 con 0x7fe2f80b1730 2024-03-25T19:44:46.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 10 monclient(hunting): my global_id is 4366 2024-03-25T19:44:46.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.372+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f80a9df0 con 0x7fe2f80b1730 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141197378 0 0) 0x7fe2f80aab70 con 0x7fe2f814be90 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 10 monclient(hunting): my global_id is 4364 2024-03-25T19:44:46.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.378+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f813bf00 con 0x7fe2f814be90 2024-03-25T19:44:46.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991684287 0 0) 0x7fe2d8004eb0 con 0x7fe2f80b4d60 2024-03-25T19:44:46.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2f80aab70 con 0x7fe2f80b4d60 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe2ec001950 con 0x7fe2f80b4d60 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.380+0000 7fe2e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:46.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.386+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008893795 0 0) 0x7fe2f80a9df0 con 0x7fe2f80b1730 2024-03-25T19:44:46.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.387+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d8004eb0 con 0x7fe2f80b1730 2024-03-25T19:44:46.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.387+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe2e8001970 con 0x7fe2f80b1730 2024-03-25T19:44:46.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.387+0000 7fe2e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.387+0000 7fe2e6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.387+0000 7fe2e6ffd640 10 monclient(hunting): dump: 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:46.387 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 96415640 0 0) 0x7fe2d8004eb0 con 0x7fe2f80b1730 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 >> v1:172.21.15.130:6790/0 conn(0x7fe2f814be90 legacy=0x7fe2f8137fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 >> v1:172.21.15.130:6789/0 conn(0x7fe2f80b4d60 legacy=0x7fe2f81d7d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 1 monclient: found mon.b 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f81d7840 con 0x7fe2f80b1730 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:44:46.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 5 monclient: authenticate success, global_id 4366 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 10 monclient: _renew_subs 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2f8138d60 con 0x7fe2f80b1730 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 10 client.4366.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 10 monclient: _renew_subs 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2f81d8650 con 0x7fe2f80b1730 2024-03-25T19:44:46.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.403+0000 7fe2fcc7b640 1 librados: init done 2024-03-25T19:44:46.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.404+0000 7fe2fcc7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.404+0000 7fe2fcc7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.404+0000 7fe2fcc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.404+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2f80627b0 con 0x7fe2f80b1730 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2e80035f0 con 0x7fe2f80b1730 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fe2e8003980 con 0x7fe2f80b1730 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2e6ffd640 10 monclient: dump: 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:46.681 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.681+0000 7fe2f4d99640 4 set_mon_vals no callback set 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.682+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fe2e8003ba0 con 0x7fe2f80b1730 2024-03-25T19:44:46.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.683+0000 7fe2f5d9b640 10 monclient: get_auth_request con 0x7fe2d806dfb0 auth_method 0 2024-03-25T19:44:46.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.683+0000 7fe2e6ffd640 10 client.4366.objecter ms_handle_connect 0x7fe2d806dfb0 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.687+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fe2e8054780 con 0x7fe2f80b1730 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.687+0000 7fe2e6ffd640 10 client.4366.objecter ms_dispatch 0x7fe2f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.687+0000 7fe2e6ffd640 3 client.4366.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.687+0000 7fe2e6ffd640 3 client.4366.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.687+0000 7fe2e6ffd640 20 client.4366.objecter dump_active .. 0 homeless 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.688+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe2e8050290 con 0x7fe2f80b1730 2024-03-25T19:44:46.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.688+0000 7fe2e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:46.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.688+0000 7fe2e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:46.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.909+0000 7fe2fcc7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.909+0000 7fe2fcc7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.909+0000 7fe2fcc7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:46.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.909+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe2f81d79d0 con 0x7fe2f80b1730 2024-03-25T19:44:46.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.994+0000 7fe2e6ffd640 1 -- 172.21.15.130:0/2726351287 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fe2e80521c0 con 0x7fe2f80b1730 2024-03-25T19:44:46.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.994+0000 7fe2e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:46.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.994+0000 7fe2e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:46.994 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:46.994 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.996+0000 7fe2fcc7b640 10 librados: watch_flush enter 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 10 librados: watch_flush exit 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 20 client.4366.objecter shutdown clearing up homeless session... 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 10 client.4366.objecter successfully canceled tick 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 >> v1:172.21.15.163:6800/1217380380 conn(0x7fe2d806dfb0 legacy=0x7fe2d8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 10 monclient: shutdown 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 >> v1:172.21.15.163:6789/0 conn(0x7fe2f80b1730 legacy=0x7fe2f80a96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 shutdown_connections 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 >> 172.21.15.130:0/2726351287 conn(0x7fe2f80617a0 msgr2=0x7fe2f8144fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:46.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 shutdown_connections 2024-03-25T19:44:46.998 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 -- 172.21.15.130:0/2726351287 wait complete. 2024-03-25T19:44:46.998 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:46.997+0000 7fe2fcc7b640 1 librados: shutdown 2024-03-25T19:44:47.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:53.817 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.933+0000 7f3ee15b5640 10 monclient: get_monmap_and_config 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.933+0000 7f3ee15b5640 10 monclient: build_initial_monmap 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.933+0000 7f3ee15b5640 10 monclient: monmap: 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:44:53.934772+0000 2024-03-25T19:44:53.933 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:44:53.934772+0000 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:44:53.934 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:53.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.934+0000 7f3ee15b5640 1 Processor -- start 2024-03-25T19:44:53.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.934+0000 7f3ee15b5640 1 -- start start 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.934+0000 7f3ee15b5640 10 monclient: init 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3edc0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3edc13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3edc1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc142520 con 0x7f3edc13f710 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc0b28a0 con 0x7f3edc0a98f0 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc0b2a30 con 0x7f3edc1430c0 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.935+0000 7f3ee15b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:53.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.936+0000 7f3ee15b5640 10 monclient(hunting): authenticate will time out at 5170.081620s 2024-03-25T19:44:53.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.949+0000 7f3eda7fc640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f3edc13f710 0x7f3edc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:41658/0 (socket says 172.21.15.130:41658) 2024-03-25T19:44:53.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.949+0000 7f3eda7fc640 1 -- 172.21.15.130:0/977973829 learned_addr learned my addr 172.21.15.130:0/977973829 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:53.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104357582 0 0) 0x7f3edc142520 con 0x7f3edc13f710 2024-03-25T19:44:53.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 10 monclient(hunting): my global_id is 4368 2024-03-25T19:44:53.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.965+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec4003520 con 0x7f3edc13f710 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3635208409 0 0) 0x7f3edc0b28a0 con 0x7f3edc0a98f0 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 10 monclient(hunting): my global_id is 4369 2024-03-25T19:44:53.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.967+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3edc142520 con 0x7f3edc0a98f0 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1536528154 0 0) 0x7f3edc0b2a30 con 0x7f3edc1430c0 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 10 monclient(hunting): my global_id is 4367 2024-03-25T19:44:53.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.975+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3edc0b28a0 con 0x7f3edc1430c0 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 712373136 0 0) 0x7f3ec4003520 con 0x7f3edc13f710 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 >> v1:172.21.15.130:6790/0 conn(0x7f3edc1430c0 legacy=0x7f3edc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 >> v1:172.21.15.163:6789/0 conn(0x7f3edc0a98f0 legacy=0x7f3edc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 monclient: found mon.noname-a 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:44:53.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc0b44c0 con 0x7f3edc13f710 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3ed0001690 con 0x7f3edc13f710 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ee15b5640 5 monclient: authenticate success, global_id 4368 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ee15b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ed9ffb640 10 monclient: dump: 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:53.981 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:53.982 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:53.982 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:53.982 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:53.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.980+0000 7f3ee15b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ed00017f0 con 0x7f3edc13f710 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 1 -- 172.21.15.130:0/977973829 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3ed0001c00 con 0x7f3edc13f710 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ee15b5640 10 monclient: get_monmap_and_config success 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ee15b5640 4 set_mon_vals no callback set 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.996+0000 7f3ed9ffb640 10 monclient: dump: 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:53.996 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:53.997 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:53.997 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:53.997 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:53.997 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 10 monclient: shutdown 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 1 -- 172.21.15.130:0/977973829 >> v1:172.21.15.130:6789/0 conn(0x7f3edc13f710 legacy=0x7f3edc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 1 -- 172.21.15.130:0/977973829 shutdown_connections 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 1 -- 172.21.15.130:0/977973829 >> 172.21.15.130:0/977973829 conn(0x7f3edc0b7ea0 msgr2=0x7f3edc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 1 -- 172.21.15.130:0/977973829 shutdown_connections 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.997+0000 7f3ee15b5640 1 -- 172.21.15.130:0/977973829 wait complete. 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 10 monclient: build_initial_monmap 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 10 monclient: monmap: 2024-03-25T19:44:53.999 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 librados: starting msgr at 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 librados: starting objecter 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 Processor -- start 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 -- start start 2024-03-25T19:44:54.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 librados: setting wanted keys 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 1 librados: calling monclient init 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.998+0000 7f3ee15b5640 10 monclient: init 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient: _reopen_session rank -1 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3edc0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-a con 0x7f3edc13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3edc1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc137f10 con 0x7f3edc13f710 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc1dc070 con 0x7f3edc0a98f0 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3edc137100 con 0x7f3edc1430c0 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient(hunting): _renew_subs 2024-03-25T19:44:54.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:53.999+0000 7f3ee15b5640 10 monclient(hunting): authenticate will time out at 5170.144990s 2024-03-25T19:44:54.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.009+0000 7f3edaffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3edc0a98f0 0x7f3edc137800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:34948/0 (socket says 172.21.15.130:34948) 2024-03-25T19:44:54.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.009+0000 7f3edaffd640 1 -- 172.21.15.130:0/1869638088 learned_addr learned my addr 172.21.15.130:0/1869638088 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:44:54.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.010+0000 7f3ec37fe640 10 client.?.objecter ms_handle_connect 0x7f3edc0a98f0 2024-03-25T19:44:54.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.010+0000 7f3ec37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.017+0000 7f3ec37fe640 10 client.?.objecter ms_handle_connect 0x7f3edc13f710 2024-03-25T19:44:54.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.017+0000 7f3ec37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1428303302 0 0) 0x7f3edc1dc070 con 0x7f3edc0a98f0 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 10 monclient(hunting): my global_id is 4372 2024-03-25T19:44:54.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.022+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ec8004eb0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3786153934 0 0) 0x7f3edc137f10 con 0x7f3edc13f710 2024-03-25T19:44:54.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:44:54.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:44:54.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:44:54.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 10 monclient(hunting): my global_id is 4371 2024-03-25T19:44:54.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.031+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3edc1dc070 con 0x7f3edc13f710 2024-03-25T19:44:54.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.032+0000 7f3ec37fe640 10 client.?.objecter ms_handle_connect 0x7f3edc1430c0 2024-03-25T19:44:54.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.032+0000 7f3ec37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2486047817 0 0) 0x7f3ec8004eb0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3edc137f10 con 0x7f3edc0a98f0 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3ecc0019f0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 10 monclient(hunting): dump: 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:54.038 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:54.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.038+0000 7f3ec37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:44:54.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1389429032 0 0) 0x7f3edc137f10 con 0x7f3edc0a98f0 2024-03-25T19:44:54.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 >> v1:172.21.15.130:6790/0 conn(0x7f3edc1430c0 legacy=0x7f3edc1db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 >> v1:172.21.15.130:6789/0 conn(0x7f3edc13f710 legacy=0x7f3edc1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 1 monclient: found mon.b 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3edc1373e0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 10 monclient: _finish_auth 0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 5 monclient: authenticate success, global_id 4372 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 10 monclient: _renew_subs 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3edc1358e0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 10 monclient: _renew_subs 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:54.047 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3edc136150 con 0x7f3edc0a98f0 2024-03-25T19:44:54.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.046+0000 7f3ee15b5640 1 librados: init done 2024-03-25T19:44:54.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.047+0000 7f3ee15b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.047+0000 7f3ee15b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.047+0000 7f3ee15b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:54.048 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.047+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3edc0627b0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ecc0035f0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3ecc003900 con 0x7f3edc0a98f0 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ec37fe640 10 monclient: dump: 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:44:54.360 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.360+0000 7f3ed97fa640 4 set_mon_vals no callback set 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.361+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f3ecc003aa0 con 0x7f3edc0a98f0 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.361+0000 7f3eda7fc640 10 monclient: get_auth_request con 0x7f3ec806dd10 auth_method 0 2024-03-25T19:44:54.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.362+0000 7f3ec37fe640 10 client.4372.objecter ms_handle_connect 0x7f3ec806dd10 2024-03-25T19:44:54.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.365+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f3ecc054830 con 0x7f3edc0a98f0 2024-03-25T19:44:54.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.365+0000 7f3ec37fe640 10 client.4372.objecter ms_dispatch 0x7f3edc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:44:54.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.365+0000 7f3ec37fe640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:44:54.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.365+0000 7f3ec37fe640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:44:54.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.365+0000 7f3ec37fe640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T19:44:54.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.366+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ecc050340 con 0x7f3edc0a98f0 2024-03-25T19:44:54.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.366+0000 7f3ec37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:44:54.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.366+0000 7f3ec37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:44:54.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.589+0000 7f3ee15b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:54.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.589+0000 7f3ee15b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:54.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.589+0000 7f3ee15b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:44:54.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.589+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3edc13e250 con 0x7f3edc0a98f0 2024-03-25T19:44:54.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.746+0000 7f3ec37fe640 1 -- 172.21.15.130:0/1869638088 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f3ecc052270 con 0x7f3edc0a98f0 2024-03-25T19:44:54.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.746+0000 7f3ec37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:44:54.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.746+0000 7f3ec37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:44:54.746 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:44:54.746 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:44:54.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.748+0000 7f3ee15b5640 10 librados: watch_flush enter 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.748+0000 7f3ee15b5640 10 librados: watch_flush exit 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.748+0000 7f3ee15b5640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.748+0000 7f3ee15b5640 10 client.4372.objecter successfully canceled tick 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.748+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 >> v1:172.21.15.163:6800/1217380380 conn(0x7f3ec806dd10 legacy=0x7f3ec80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 10 monclient: shutdown 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 >> v1:172.21.15.163:6789/0 conn(0x7f3edc0a98f0 legacy=0x7f3edc137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 shutdown_connections 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 >> 172.21.15.130:0/1869638088 conn(0x7f3edc0b7ea0 msgr2=0x7f3edc14ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 shutdown_connections 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 -- 172.21.15.130:0/1869638088 wait complete. 2024-03-25T19:44:54.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:44:54.749+0000 7f3ee15b5640 1 librados: shutdown 2024-03-25T19:44:54.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:44:59.895 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:44.828 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.908 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:44.841 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.948 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:03:44.881 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.957 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:44.858 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.963 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:44.864 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.968 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:44.901 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.976 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:44.909 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:44:59.990 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:03:44.891 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:00.016 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:03:44.949 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:01.567 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.683+0000 7f29655ac640 10 monclient: get_monmap_and_config 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.683+0000 7f29655ac640 10 monclient: build_initial_monmap 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.683+0000 7f29655ac640 10 monclient: monmap: 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:01.684232+0000 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:01.684232+0000 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:01.683 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:01.684 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:01.684 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:01.684 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.684+0000 7f29655ac640 1 Processor -- start 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.684+0000 7f29655ac640 1 -- start start 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.684+0000 7f29655ac640 10 monclient: init 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-a con 0x7f296006ab80 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:01.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-c con 0x7f296006af60 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-b con 0x7f29600b00b0 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29600b0500 con 0x7f296006ab80 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296013ffa0 con 0x7f296006af60 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29600af520 con 0x7f29600b00b0 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:01.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.685+0000 7f29655ac640 10 monclient(hunting): authenticate will time out at 5177.830772s 2024-03-25T19:45:01.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.694+0000 7f2957fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f296006af60 0x7f296013f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:53492/0 (socket says 172.21.15.130:53492) 2024-03-25T19:45:01.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.694+0000 7f2957fff640 1 -- 172.21.15.130:0/201752324 learned_addr learned my addr 172.21.15.130:0/201752324 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804378837 0 0) 0x7f29600b0500 con 0x7f296006ab80 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4374 2024-03-25T19:45:01.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.704+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c003520 con 0x7f296006ab80 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430339478 0 0) 0x7f296013ffa0 con 0x7f296006af60 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4375 2024-03-25T19:45:01.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.709+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29600b0500 con 0x7f296006af60 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103726391 0 0) 0x7f29600af520 con 0x7f29600b00b0 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T19:45:01.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.722+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296013ffa0 con 0x7f29600b00b0 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607110775 0 0) 0x7f29600b0500 con 0x7f296006af60 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 >> v1:172.21.15.130:6790/0 conn(0x7f29600b00b0 legacy=0x7f296014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 >> v1:172.21.15.130:6789/0 conn(0x7f296006ab80 legacy=0x7f29600ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 monclient: found mon.noname-c 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2960072890 con 0x7f296006af60 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2948001670 con 0x7f296006af60 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f29655ac640 5 monclient: authenticate success, global_id 4375 2024-03-25T19:45:01.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f295e7fc640 10 monclient: dump: 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.725 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.724+0000 7f29655ac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29480017d0 con 0x7f296006af60 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 1 -- 172.21.15.130:0/201752324 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2948001be0 con 0x7f296006af60 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f29655ac640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f29655ac640 4 set_mon_vals no callback set 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.732+0000 7f295e7fc640 10 monclient: dump: 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.732 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.733 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:01.733 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:01.733 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:01.733 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:01.733 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 10 monclient: shutdown 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 1 -- 172.21.15.130:0/201752324 >> v1:172.21.15.163:6789/0 conn(0x7f296006af60 legacy=0x7f296013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 1 -- 172.21.15.130:0/201752324 shutdown_connections 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 1 -- 172.21.15.130:0/201752324 >> 172.21.15.130:0/201752324 conn(0x7f29600b4a80 msgr2=0x7f29600b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 1 -- 172.21.15.130:0/201752324 shutdown_connections 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.733+0000 7f29655ac640 1 -- 172.21.15.130:0/201752324 wait complete. 2024-03-25T19:45:01.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 10 monclient: build_initial_monmap 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 10 monclient: monmap: 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 librados: starting msgr at 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 librados: starting objecter 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 Processor -- start 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 -- start start 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 librados: setting wanted keys 2024-03-25T19:45:01.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 1 librados: calling monclient init 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.734+0000 7f29655ac640 10 monclient: init 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-a con 0x7f296006ab80 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-b con 0x7f29600b00b0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient(hunting): picked mon.noname-c con 0x7f29601db620 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2960144d80 con 0x7f296006ab80 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2960144ff0 con 0x7f29600b00b0 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29601da9f0 con 0x7f29601db620 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:01.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.735+0000 7f29655ac640 10 monclient(hunting): authenticate will time out at 5177.880946s 2024-03-25T19:45:01.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.744+0000 7f295f7fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f29601db620 0x7f2960145dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:39996/0 (socket says 172.21.15.130:39996) 2024-03-25T19:45:01.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.744+0000 7f295f7fe640 1 -- 172.21.15.130:0/2627582645 learned_addr learned my addr 172.21.15.130:0/2627582645 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:01.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.744+0000 7f29577fe640 10 client.?.objecter ms_handle_connect 0x7f29601db620 2024-03-25T19:45:01.744 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.744+0000 7f29577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538696965 0 0) 0x7f29601da9f0 con 0x7f29601db620 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 10 monclient(hunting): my global_id is 4376 2024-03-25T19:45:01.750 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.750+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2940004eb0 con 0x7f29601db620 2024-03-25T19:45:01.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.756+0000 7f29577fe640 10 client.?.objecter ms_handle_connect 0x7f29600b00b0 2024-03-25T19:45:01.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.756+0000 7f29577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:01.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.758+0000 7f29577fe640 10 client.?.objecter ms_handle_connect 0x7f296006ab80 2024-03-25T19:45:01.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.758+0000 7f29577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1291349913 0 0) 0x7f2940004eb0 con 0x7f29601db620 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29601da9f0 con 0x7f29601db620 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2958001d00 con 0x7f29601db620 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 10 monclient(hunting): dump: 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.764 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.764+0000 7f29577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:01.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111831399 0 0) 0x7f2960144d80 con 0x7f296006ab80 2024-03-25T19:45:01.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T19:45:01.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.767+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2940004eb0 con 0x7f296006ab80 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759184589 0 0) 0x7f2960144ff0 con 0x7f29600b00b0 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 10 monclient(hunting): my global_id is 4378 2024-03-25T19:45:01.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.776+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2960144d80 con 0x7f29600b00b0 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3475565341 0 0) 0x7f2940004eb0 con 0x7f296006ab80 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2960144ff0 con 0x7f296006ab80 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f29500017b0 con 0x7f296006ab80 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.777+0000 7f29577fe640 10 monclient(hunting): dump: 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:01.777 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:01.778 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:01.778 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:01.778 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2211676819 0 0) 0x7f29601da9f0 con 0x7f29601db620 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 >> v1:172.21.15.163:6789/0 conn(0x7f29600b00b0 legacy=0x7f29601daf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 >> v1:172.21.15.130:6789/0 conn(0x7f296006ab80 legacy=0x7f2960143c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 1 monclient: found mon.c 2024-03-25T19:45:01.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29601464d0 con 0x7f29601db620 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 5 monclient: authenticate success, global_id 4376 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 10 monclient: _renew_subs 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2960146720 con 0x7f29601db620 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 10 monclient: _renew_subs 2024-03-25T19:45:01.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29601458d0 con 0x7f29601db620 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.785+0000 7f29655ac640 1 librados: init done 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.786+0000 7f29655ac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.786+0000 7f29655ac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.786+0000 7f29655ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:01.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:01.786+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29601dab80 con 0x7f29601db620 2024-03-25T19:45:02.108 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2958003990 con 0x7f29601db620 2024-03-25T19:45:02.108 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2958003d60 con 0x7f29601db620 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 10 monclient: dump: 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:02.109 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:02.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f295dffb640 4 set_mon_vals no callback set 2024-03-25T19:45:02.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.109+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f2958003f80 con 0x7f29601db620 2024-03-25T19:45:02.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.110+0000 7f295effd640 10 monclient: get_auth_request con 0x7f294006e020 auth_method 0 2024-03-25T19:45:02.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.110+0000 7f29577fe640 10 client.4376.objecter ms_handle_connect 0x7f294006e020 2024-03-25T19:45:02.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f2958054b60 con 0x7f29601db620 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 10 client.4376.objecter ms_dispatch 0x7f2960000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2958050670 con 0x7f29601db620 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:02.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.115+0000 7f29577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:02.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.350+0000 7f29655ac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.350+0000 7f29655ac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.350+0000 7f29655ac640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:02.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.350+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29600627b0 con 0x7f29601db620 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.388+0000 7f29577fe640 1 -- 172.21.15.130:0/2627582645 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f29580525a0 con 0x7f29601db620 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.388+0000 7f29577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:02.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.388+0000 7f29577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:02.389 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 10 librados: watch_flush enter 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 10 librados: watch_flush exit 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 10 client.4376.objecter successfully canceled tick 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 >> v1:172.21.15.163:6800/1217380380 conn(0x7f294006e020 legacy=0x7f29400704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 10 monclient: shutdown 2024-03-25T19:45:02.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 >> v1:172.21.15.130:6790/0 conn(0x7f29601db620 legacy=0x7f2960145dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 shutdown_connections 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 >> 172.21.15.130:0/2627582645 conn(0x7f29600b4a80 msgr2=0x7f29600ad880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 shutdown_connections 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 -- 172.21.15.130:0/2627582645 wait complete. 2024-03-25T19:45:02.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:02.391+0000 7f29655ac640 1 librados: shutdown 2024-03-25T19:45:02.404 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:09.209 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.326+0000 7f427ed3d640 10 monclient: get_monmap_and_config 2024-03-25T19:45:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.326+0000 7f427ed3d640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.326+0000 7f427ed3d640 10 monclient: monmap: 2024-03-25T19:45:09.326 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:09.326 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:09.327691+0000 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:09.327691+0000 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:09.327 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.327+0000 7f427ed3d640 1 Processor -- start 2024-03-25T19:45:09.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.327+0000 7f427ed3d640 1 -- start start 2024-03-25T19:45:09.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.327+0000 7f427ed3d640 10 monclient: init 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-a con 0x7f42780a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-b con 0x7f427813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-c con 0x7f42781430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4278142520 con 0x7f42780a98f0 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42780b28a0 con 0x7f42781430c0 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42780b2a30 con 0x7f427813f710 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.328+0000 7f427ed3d640 10 monclient(hunting): authenticate will time out at 5185.474123s 2024-03-25T19:45:09.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.335+0000 7f4277fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f427813f710 0x7f42781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:36512/0 (socket says 172.21.15.130:36512) 2024-03-25T19:45:09.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.335+0000 7f4277fff640 1 -- 172.21.15.130:0/1327016949 learned_addr learned my addr 172.21.15.130:0/1327016949 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261066962 0 0) 0x7f42780b2a30 con 0x7f427813f710 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 10 monclient(hunting): my global_id is 4379 2024-03-25T19:45:09.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.345+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4268003520 con 0x7f427813f710 2024-03-25T19:45:09.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706078117 0 0) 0x7f42780b28a0 con 0x7f42781430c0 2024-03-25T19:45:09.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T19:45:09.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.354+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42780b2a30 con 0x7f42781430c0 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 291046455 0 0) 0x7f4268003520 con 0x7f427813f710 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 >> v1:172.21.15.163:6789/0 conn(0x7f42781430c0 legacy=0x7f427814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 >> v1:172.21.15.130:6789/0 conn(0x7f42780a98f0 legacy=0x7f42780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 monclient: found mon.noname-b 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42780b44c0 con 0x7f427813f710 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4264001690 con 0x7f427813f710 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f427ed3d640 5 monclient: authenticate success, global_id 4379 2024-03-25T19:45:09.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f427ed3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f42777fe640 10 monclient: dump: 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:09.362 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.363 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.361+0000 7f427ed3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42640017f0 con 0x7f427813f710 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 1 -- 172.21.15.130:0/1327016949 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4264001c00 con 0x7f427813f710 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f427ed3d640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f427ed3d640 4 set_mon_vals no callback set 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.376+0000 7f42777fe640 10 monclient: dump: 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:09.376 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:09.377 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.377+0000 7f427ed3d640 10 monclient: shutdown 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.377+0000 7f427ed3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.377+0000 7f427ed3d640 1 -- 172.21.15.130:0/1327016949 >> v1:172.21.15.130:6790/0 conn(0x7f427813f710 legacy=0x7f42781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.377+0000 7f427ed3d640 1 -- 172.21.15.130:0/1327016949 shutdown_connections 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.377+0000 7f427ed3d640 1 -- 172.21.15.130:0/1327016949 >> 172.21.15.130:0/1327016949 conn(0x7f42780b7ea0 msgr2=0x7f42780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.378+0000 7f427ed3d640 1 -- 172.21.15.130:0/1327016949 shutdown_connections 2024-03-25T19:45:09.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.378+0000 7f427ed3d640 1 -- 172.21.15.130:0/1327016949 wait complete. 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 10 monclient: build_initial_monmap 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 10 monclient: monmap: 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 librados: starting msgr at 2024-03-25T19:45:09.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 librados: starting objecter 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 Processor -- start 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 -- start start 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 librados: setting wanted keys 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 1 librados: calling monclient init 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.379+0000 7f427ed3d640 10 monclient: init 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-b con 0x7f42780a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-c con 0x7f427813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient(hunting): picked mon.noname-a con 0x7f42781430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4278137df0 con 0x7f42781430c0 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42781337a0 con 0x7f42780a98f0 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4278133930 con 0x7f427813f710 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:09.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.380+0000 7f427ed3d640 10 monclient(hunting): authenticate will time out at 5185.526082s 2024-03-25T19:45:09.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.393+0000 7f427d2da640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f42781430c0 0x7f4278133090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:45926/0 (socket says 172.21.15.130:45926) 2024-03-25T19:45:09.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.393+0000 7f427d2da640 1 -- 172.21.15.130:0/3236187163 learned_addr learned my addr 172.21.15.130:0/3236187163 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:09.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.397+0000 7f42757fa640 10 client.?.objecter ms_handle_connect 0x7f42781430c0 2024-03-25T19:45:09.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.397+0000 7f42757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.403+0000 7f42757fa640 10 client.?.objecter ms_handle_connect 0x7f427813f710 2024-03-25T19:45:09.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.403+0000 7f42757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.408+0000 7f42757fa640 10 client.?.objecter ms_handle_connect 0x7f42780a98f0 2024-03-25T19:45:09.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.408+0000 7f42757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3092784908 0 0) 0x7f4278133930 con 0x7f427813f710 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 10 monclient(hunting): my global_id is 4382 2024-03-25T19:45:09.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.411+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f425c004eb0 con 0x7f427813f710 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3803219257 0 0) 0x7f4278137df0 con 0x7f42781430c0 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 10 monclient(hunting): my global_id is 4383 2024-03-25T19:45:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.412+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4278133930 con 0x7f42781430c0 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2346446622 0 0) 0x7f42781337a0 con 0x7f42780a98f0 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T19:45:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.415+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4278137df0 con 0x7f42780a98f0 2024-03-25T19:45:09.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2146114631 0 0) 0x7f425c004eb0 con 0x7f427813f710 2024-03-25T19:45:09.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42781337a0 con 0x7f427813f710 2024-03-25T19:45:09.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4264001bc0 con 0x7f427813f710 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 10 monclient(hunting): dump: 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:09.424 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:09.425 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.424+0000 7f42757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:09.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1492401834 0 0) 0x7f42781337a0 con 0x7f427813f710 2024-03-25T19:45:09.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 >> v1:172.21.15.163:6789/0 conn(0x7f42780a98f0 legacy=0x7f42781392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 >> v1:172.21.15.130:6789/0 conn(0x7f42781430c0 legacy=0x7f4278133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:09.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:09.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 1 monclient: found mon.c 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42781367d0 con 0x7f427813f710 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f42757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 5 monclient: authenticate success, global_id 4382 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 10 monclient: _renew_subs 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42781369e0 con 0x7f427813f710 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 10 monclient: _renew_subs 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f427813b640 con 0x7f427813f710 2024-03-25T19:45:09.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.429+0000 7f427ed3d640 1 librados: init done 2024-03-25T19:45:09.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.430+0000 7f427ed3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.430+0000 7f427ed3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.430+0000 7f427ed3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.430+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42780627b0 con 0x7f427813f710 2024-03-25T19:45:09.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.720+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4264004a90 con 0x7f427813f710 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f42757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4264001250 con 0x7f427813f710 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f42757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f42757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f42757fa640 10 monclient: dump: 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:09.721 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:09.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.721+0000 7f4276ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:09.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.722+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f426401acd0 con 0x7f427813f710 2024-03-25T19:45:09.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.722+0000 7f427cad9640 10 monclient: get_auth_request con 0x7f425c06ddf0 auth_method 0 2024-03-25T19:45:09.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.723+0000 7f42757fa640 10 client.4382.objecter ms_handle_connect 0x7f425c06ddf0 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f4264054910 con 0x7f427813f710 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 10 client.4382.objecter ms_dispatch 0x7f4278000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4264050250 con 0x7f427813f710 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:09.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.727+0000 7f42757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.978+0000 7f427ed3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.978+0000 7f427ed3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.978+0000 7f427ed3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:09.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:09.978+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42780008d0 con 0x7f427813f710 2024-03-25T19:45:10.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.126+0000 7f42757fa640 1 -- 172.21.15.130:0/3236187163 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f4264052350 con 0x7f427813f710 2024-03-25T19:45:10.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.126+0000 7f42757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:10.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.126+0000 7f42757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:10.126 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:10.126 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 10 librados: watch_flush enter 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 10 librados: watch_flush exit 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 10 client.4382.objecter successfully canceled tick 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 >> v1:172.21.15.163:6800/1217380380 conn(0x7f425c06ddf0 legacy=0x7f425c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 10 monclient: shutdown 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.128+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 >> v1:172.21.15.130:6790/0 conn(0x7f427813f710 legacy=0x7f42781376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.129+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 shutdown_connections 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.129+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 >> 172.21.15.130:0/3236187163 conn(0x7f42780b7ea0 msgr2=0x7f4278074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.129+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 shutdown_connections 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.129+0000 7f427ed3d640 1 -- 172.21.15.130:0/3236187163 wait complete. 2024-03-25T19:45:10.129 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:10.129+0000 7f427ed3d640 1 librados: shutdown 2024-03-25T19:45:10.140 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:16.946 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.064+0000 7f388c0e6640 10 monclient: get_monmap_and_config 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.064+0000 7f388c0e6640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.064+0000 7f388c0e6640 10 monclient: monmap: 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:17.066145+0000 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:17.066145+0000 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:17.065 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.065+0000 7f388c0e6640 1 Processor -- start 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.065+0000 7f388c0e6640 1 -- start start 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.065+0000 7f388c0e6640 10 monclient: init 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-a con 0x7f38840b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:17.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-c con 0x7f38840b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-b con 0x7f388414be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840af940 con 0x7f38840b4d60 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840b0a70 con 0x7f38840b1730 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38840b0d50 con 0x7f388414be90 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.066+0000 7f388c0e6640 10 monclient(hunting): authenticate will time out at 5193.212861s 2024-03-25T19:45:17.074 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.074+0000 7f388a683640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f388414be90 0x7f388414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:54888/0 (socket says 172.21.15.130:54888) 2024-03-25T19:45:17.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.074+0000 7f388a683640 1 -- 172.21.15.130:0/1941851327 learned_addr learned my addr 172.21.15.130:0/1941851327 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:17.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2185532647 0 0) 0x7f38840af940 con 0x7f38840b4d60 2024-03-25T19:45:17.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): my global_id is 4386 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3870003520 con 0x7f38840b4d60 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1798869242 0 0) 0x7f38840b0d50 con 0x7f388414be90 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 10 monclient(hunting): my global_id is 4385 2024-03-25T19:45:17.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.084+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38840af940 con 0x7f388414be90 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2637247449 0 0) 0x7f38840af940 con 0x7f388414be90 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 >> v1:172.21.15.163:6789/0 conn(0x7f38840b1730 legacy=0x7f38840b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 >> v1:172.21.15.130:6789/0 conn(0x7f38840b4d60 legacy=0x7f388413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 monclient: found mon.noname-b 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38840b1030 con 0x7f388414be90 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f387c0019f0 con 0x7f388414be90 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f3888e80640 10 monclient: dump: 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f388c0e6640 5 monclient: authenticate success, global_id 4385 2024-03-25T19:45:17.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.097+0000 7f388c0e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:17.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f387c001b50 con 0x7f388414be90 2024-03-25T19:45:17.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 1 -- 172.21.15.130:0/1941851327 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f387c001f20 con 0x7f388414be90 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f388c0e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f388c0e6640 4 set_mon_vals no callback set 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.110+0000 7f3888e80640 10 monclient: dump: 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.111 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:17.112 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:17.112 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:17.112 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:17.112 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 10 monclient: shutdown 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 1 -- 172.21.15.130:0/1941851327 >> v1:172.21.15.130:6790/0 conn(0x7f388414be90 legacy=0x7f388414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 1 -- 172.21.15.130:0/1941851327 shutdown_connections 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 1 -- 172.21.15.130:0/1941851327 >> 172.21.15.130:0/1941851327 conn(0x7f38840617a0 msgr2=0x7f3884061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 1 -- 172.21.15.130:0/1941851327 shutdown_connections 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.111+0000 7f388c0e6640 1 -- 172.21.15.130:0/1941851327 wait complete. 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 10 monclient: build_initial_monmap 2024-03-25T19:45:17.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 10 monclient: monmap: 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 librados: starting msgr at 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 librados: starting objecter 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 Processor -- start 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 -- start start 2024-03-25T19:45:17.114 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 librados: setting wanted keys 2024-03-25T19:45:17.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 1 librados: calling monclient init 2024-03-25T19:45:17.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.112+0000 7f388c0e6640 10 monclient: init 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-a con 0x7f38840b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-c con 0x7f38840b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 10 monclient(hunting): picked mon.noname-b con 0x7f388414be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3884138340 con 0x7f38840b1730 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3884137a50 con 0x7f388414be90 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.113+0000 7f388c0e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3884137be0 con 0x7f38840b4d60 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.114+0000 7f388c0e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:17.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.114+0000 7f388c0e6640 10 monclient(hunting): authenticate will time out at 5193.260646s 2024-03-25T19:45:17.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.117+0000 7f3889681640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f38840b4d60 0x7f38841dab90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:54912/0 (socket says 172.21.15.130:54912) 2024-03-25T19:45:17.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.117+0000 7f3889681640 1 -- 172.21.15.130:0/2847288149 learned_addr learned my addr 172.21.15.130:0/2847288149 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:17.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.118+0000 7f387a7fc640 10 client.?.objecter ms_handle_connect 0x7f38840b4d60 2024-03-25T19:45:17.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.118+0000 7f387a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.119+0000 7f387a7fc640 10 client.?.objecter ms_handle_connect 0x7f388414be90 2024-03-25T19:45:17.120 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.119+0000 7f387a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.122+0000 7f387a7fc640 10 client.?.objecter ms_handle_connect 0x7f38840b1730 2024-03-25T19:45:17.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.122+0000 7f387a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:17.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048080029 0 0) 0x7f3884137be0 con 0x7f38840b4d60 2024-03-25T19:45:17.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 10 monclient(hunting): my global_id is 4388 2024-03-25T19:45:17.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.124+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3868004eb0 con 0x7f38840b4d60 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626525361 0 0) 0x7f3884138340 con 0x7f38840b1730 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 10 monclient(hunting): my global_id is 4389 2024-03-25T19:45:17.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.132+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3884137be0 con 0x7f38840b1730 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410108714 0 0) 0x7f3884137a50 con 0x7f388414be90 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 10 monclient(hunting): my global_id is 4390 2024-03-25T19:45:17.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.134+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3884138340 con 0x7f388414be90 2024-03-25T19:45:17.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4031003940 0 0) 0x7f3868004eb0 con 0x7f38840b4d60 2024-03-25T19:45:17.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3884137a50 con 0x7f38840b4d60 2024-03-25T19:45:17.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f38740019d0 con 0x7f38840b4d60 2024-03-25T19:45:17.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.138+0000 7f387a7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.139+0000 7f387a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:17.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 550013367 0 0) 0x7f3884137a50 con 0x7f38840b4d60 2024-03-25T19:45:17.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 >> v1:172.21.15.163:6789/0 conn(0x7f388414be90 legacy=0x7f38841e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 >> v1:172.21.15.130:6789/0 conn(0x7f38840b1730 legacy=0x7f3884131d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 1 monclient: found mon.c 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3884140380 con 0x7f38840b4d60 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f387a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f388c0e6640 5 monclient: authenticate success, global_id 4388 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f388c0e6640 10 monclient: _renew_subs 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f388c0e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.146+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38841405d0 con 0x7f38840b4d60 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.147+0000 7f388c0e6640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.147+0000 7f388c0e6640 10 monclient: _renew_subs 2024-03-25T19:45:17.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.147+0000 7f388c0e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.147+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3884144f20 con 0x7f38840b4d60 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.147+0000 7f388c0e6640 1 librados: init done 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.148+0000 7f388c0e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.148+0000 7f388c0e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.148+0000 7f388c0e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.148+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38840627b0 con 0x7f38840b4d60 2024-03-25T19:45:17.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38740035f0 con 0x7f38840b4d60 2024-03-25T19:45:17.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:17.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3874003a00 con 0x7f38840b4d60 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387a7fc640 10 monclient: dump: 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:17.439 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:17.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.437+0000 7f387bfff640 4 set_mon_vals no callback set 2024-03-25T19:45:17.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.438+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f3874003c20 con 0x7f38840b4d60 2024-03-25T19:45:17.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.439+0000 7f3889e82640 10 monclient: get_auth_request con 0x7f386806ddf0 auth_method 0 2024-03-25T19:45:17.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.439+0000 7f387a7fc640 10 client.4388.objecter ms_handle_connect 0x7f386806ddf0 2024-03-25T19:45:17.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.443+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f38740547d0 con 0x7f38840b4d60 2024-03-25T19:45:17.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.443+0000 7f387a7fc640 10 client.4388.objecter ms_dispatch 0x7f3884000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:17.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.443+0000 7f387a7fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:17.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.443+0000 7f387a7fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:17.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.444+0000 7f387a7fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T19:45:17.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.444+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38740502e0 con 0x7f38840b4d60 2024-03-25T19:45:17.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.444+0000 7f387a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:17.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.444+0000 7f387a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:17.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.667+0000 7f388c0e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.667+0000 7f388c0e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.667+0000 7f388c0e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:17.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.667+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3884140760 con 0x7f38840b4d60 2024-03-25T19:45:17.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.688+0000 7f387a7fc640 1 -- 172.21.15.130:0/2847288149 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f3874052210 con 0x7f38840b4d60 2024-03-25T19:45:17.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.688+0000 7f387a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:17.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.688+0000 7f387a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:17.689 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:17.690 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:17.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 10 librados: watch_flush enter 2024-03-25T19:45:17.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 10 librados: watch_flush exit 2024-03-25T19:45:17.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T19:45:17.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 10 client.4388.objecter successfully canceled tick 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 >> v1:172.21.15.163:6800/1217380380 conn(0x7f386806ddf0 legacy=0x7f38680702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 10 monclient: shutdown 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 >> v1:172.21.15.130:6790/0 conn(0x7f38840b4d60 legacy=0x7f38841dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 shutdown_connections 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.691+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 >> 172.21.15.130:0/2847288149 conn(0x7f38840617a0 msgr2=0x7f38840b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.692+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 shutdown_connections 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.692+0000 7f388c0e6640 1 -- 172.21.15.130:0/2847288149 wait complete. 2024-03-25T19:45:17.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:17.692+0000 7f388c0e6640 1 librados: shutdown 2024-03-25T19:45:17.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:19.988 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:04.921 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:19.995 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:04.928 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.035 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:04.969 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.042 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:04.943 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.050 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:04.951 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.079 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:05.012 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.079 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:04.981 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.088 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:05.021 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:20.115 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:05.048 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:45:24.512 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:24.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 10 monclient: get_monmap_and_config 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 10 monclient: build_initial_monmap 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 10 monclient: monmap: 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:24.626926+0000 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:24.626926+0000 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:24.626 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:24.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 1 Processor -- start 2024-03-25T19:45:24.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 1 -- start start 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.626+0000 7f8864e25640 10 monclient: init 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-b con 0x7f88600a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-a con 0x7f886013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-c con 0x7f88601430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8860142520 con 0x7f886013f710 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600b28a0 con 0x7f88601430c0 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600b2a30 con 0x7f88600a98f0 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:24.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.627+0000 7f8864e25640 10 monclient(hunting): authenticate will time out at 5200.773208s 2024-03-25T19:45:24.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.635+0000 7f8855d9b640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f886013f710 0x7f88601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:55516/0 (socket says 172.21.15.130:55516) 2024-03-25T19:45:24.635 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.635+0000 7f8855d9b640 1 -- 172.21.15.130:0/980297094 learned_addr learned my addr 172.21.15.130:0/980297094 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:24.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609374318 0 0) 0x7f88600b28a0 con 0x7f88601430c0 2024-03-25T19:45:24.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 10 monclient(hunting): my global_id is 4393 2024-03-25T19:45:24.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.652+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f884c003520 con 0x7f88601430c0 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3603726571 0 0) 0x7f8860142520 con 0x7f886013f710 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 10 monclient(hunting): my global_id is 4392 2024-03-25T19:45:24.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.657+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88600b28a0 con 0x7f886013f710 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478331886 0 0) 0x7f88600b2a30 con 0x7f88600a98f0 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 10 monclient(hunting): my global_id is 4391 2024-03-25T19:45:24.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.660+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860142520 con 0x7f88600a98f0 2024-03-25T19:45:24.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1444273360 0 0) 0x7f88600b28a0 con 0x7f886013f710 2024-03-25T19:45:24.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 >> v1:172.21.15.130:6790/0 conn(0x7f88600a98f0 legacy=0x7f88600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 >> v1:172.21.15.163:6789/0 conn(0x7f88601430c0 legacy=0x7f886014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 10 monclient: _finish_hunting 0 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 1 monclient: found mon.noname-a 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.667+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600b44c0 con 0x7f886013f710 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 10 monclient: _finish_auth 0 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 10 monclient: _check_auth_tickets 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8848001670 con 0x7f886013f710 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f885dd9b640 10 monclient: dump: 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:24.668 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f8864e25640 5 monclient: authenticate success, global_id 4392 2024-03-25T19:45:24.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.668+0000 7f8864e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.680+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88480017d0 con 0x7f886013f710 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.680+0000 7f885dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f885dd9b640 1 -- 172.21.15.130:0/980297094 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8848001be0 con 0x7f886013f710 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 4 set_mon_vals no callback set 2024-03-25T19:45:24.680 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f885dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f885dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f885dd9b640 10 monclient: dump: 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:24.681 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:24.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 10 monclient: shutdown 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 1 -- 172.21.15.130:0/980297094 >> v1:172.21.15.130:6789/0 conn(0x7f886013f710 legacy=0x7f88601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 1 -- 172.21.15.130:0/980297094 shutdown_connections 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 1 -- 172.21.15.130:0/980297094 >> 172.21.15.130:0/980297094 conn(0x7f88600b7ea0 msgr2=0x7f88600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 1 -- 172.21.15.130:0/980297094 shutdown_connections 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.681+0000 7f8864e25640 1 -- 172.21.15.130:0/980297094 wait complete. 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.682+0000 7f8864e25640 10 monclient: build_initial_monmap 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.682+0000 7f8864e25640 10 monclient: monmap: 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:24.684 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 librados: starting msgr at 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 librados: starting objecter 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 Processor -- start 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 -- start start 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 librados: setting wanted keys 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 1 librados: calling monclient init 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 10 monclient: init 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:24.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-c con 0x7f88600a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.683+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-a con 0x7f886013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 10 monclient(hunting): picked mon.noname-b con 0x7f88601430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886013c530 con 0x7f886013f710 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886012f860 con 0x7f88601430c0 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f886012fad0 con 0x7f88600a98f0 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:24.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.684+0000 7f8864e25640 10 monclient(hunting): authenticate will time out at 5200.829720s 2024-03-25T19:45:24.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.692+0000 7f885ed9d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f88601430c0 0x7f88601e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:39732/0 (socket says 172.21.15.130:39732) 2024-03-25T19:45:24.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.692+0000 7f885ed9d640 1 -- 172.21.15.130:0/53886467 learned_addr learned my addr 172.21.15.130:0/53886467 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:24.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.694+0000 7f88577fe640 10 client.?.objecter ms_handle_connect 0x7f88601430c0 2024-03-25T19:45:24.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.694+0000 7f88577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.702+0000 7f88577fe640 10 client.?.objecter ms_handle_connect 0x7f88600a98f0 2024-03-25T19:45:24.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.702+0000 7f88577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.704+0000 7f88577fe640 10 client.?.objecter ms_handle_connect 0x7f886013f710 2024-03-25T19:45:24.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.704+0000 7f88577fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:24.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2220299024 0 0) 0x7f886012fad0 con 0x7f88600a98f0 2024-03-25T19:45:24.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 10 monclient(hunting): my global_id is 4394 2024-03-25T19:45:24.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.705+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8840004eb0 con 0x7f88600a98f0 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540556928 0 0) 0x7f886012f860 con 0x7f88601430c0 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T19:45:24.711 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.712+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886012fad0 con 0x7f88601430c0 2024-03-25T19:45:24.713 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.714+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3852972341 0 0) 0x7f8840004eb0 con 0x7f88600a98f0 2024-03-25T19:45:24.713 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.714+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f886012f860 con 0x7f88600a98f0 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.715+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f88500019b0 con 0x7f88600a98f0 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.715+0000 7f88577fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.715+0000 7f88577fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.716+0000 7f88577fe640 10 monclient(hunting): dump: 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:24.715 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:24.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.716+0000 7f88577fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:24.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.719+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2381190423 0 0) 0x7f886012f860 con 0x7f88600a98f0 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.719+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 >> v1:172.21.15.163:6789/0 conn(0x7f88601430c0 legacy=0x7f88601e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 >> v1:172.21.15.130:6789/0 conn(0x7f886013f710 legacy=0x7f88601df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 1 monclient: found mon.c 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8860131320 con 0x7f88600a98f0 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f88577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 5 monclient: authenticate success, global_id 4394 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 10 monclient: _renew_subs 2024-03-25T19:45:24.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f886013c860 con 0x7f88600a98f0 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 10 client.4394.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 10 monclient: _renew_subs 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f886013bf20 con 0x7f88600a98f0 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.720+0000 7f8864e25640 1 librados: init done 2024-03-25T19:45:24.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.721+0000 7f8864e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:24.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.721+0000 7f8864e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:24.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.721+0000 7f8864e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:24.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:24.721+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88600627b0 con 0x7f88600a98f0 2024-03-25T19:45:25.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.003+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8850003690 con 0x7f88600a98f0 2024-03-25T19:45:25.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.003+0000 7f88577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:25.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.003+0000 7f885d59a640 4 set_mon_vals no callback set 2024-03-25T19:45:25.016 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.017+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8850003a60 con 0x7f88600a98f0 2024-03-25T19:45:25.016 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.017+0000 7f88577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.017+0000 7f88577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.017+0000 7f88577fe640 10 monclient: dump: 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:25.017 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:25.018 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.018+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8850003c40 con 0x7f88600a98f0 2024-03-25T19:45:25.018 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.018+0000 7f8855d9b640 10 monclient: get_auth_request con 0x7f884006dc80 auth_method 0 2024-03-25T19:45:25.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.018+0000 7f88577fe640 10 client.4394.objecter ms_handle_connect 0x7f884006dc80 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8850053720 con 0x7f88600a98f0 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 10 client.4394.objecter ms_dispatch 0x7f8860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 3 client.4394.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 3 client.4394.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 20 client.4394.objecter dump_active .. 0 homeless 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f885004f230 con 0x7f88600a98f0 2024-03-25T19:45:25.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:25.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.025+0000 7f88577fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:25.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.277+0000 7f8864e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.277+0000 7f8864e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.277+0000 7f8864e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:45:25.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.277+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8860069f40 con 0x7f88600a98f0 2024-03-25T19:45:25.302 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.302+0000 7f88577fe640 1 -- 172.21.15.130:0/53886467 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8850051160 con 0x7f88600a98f0 2024-03-25T19:45:25.302 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.302+0000 7f88577fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:25.302 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.302+0000 7f88577fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:25.302 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:25.302 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:25.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 10 librados: watch_flush enter 2024-03-25T19:45:25.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 10 librados: watch_flush exit 2024-03-25T19:45:25.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 20 client.4394.objecter shutdown clearing up homeless session... 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 10 client.4394.objecter successfully canceled tick 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 >> v1:172.21.15.163:6800/1217380380 conn(0x7f884006dc80 legacy=0x7f8840070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 10 monclient: shutdown 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 >> v1:172.21.15.130:6790/0 conn(0x7f88600a98f0 legacy=0x7f88600b7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 shutdown_connections 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 >> 172.21.15.130:0/53886467 conn(0x7f88600b7ea0 msgr2=0x7f8860074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 shutdown_connections 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 -- 172.21.15.130:0/53886467 wait complete. 2024-03-25T19:45:25.306 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:25.305+0000 7f8864e25640 1 librados: shutdown 2024-03-25T19:45:25.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:32.126 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.243+0000 7f11e37a3640 10 monclient: get_monmap_and_config 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.243+0000 7f11e37a3640 10 monclient: build_initial_monmap 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.243+0000 7f11e37a3640 10 monclient: monmap: 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:32.244541+0000 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:32.244541+0000 2024-03-25T19:45:32.243 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:32.244 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.244 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:32.244 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:32.244 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:32.244 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.244+0000 7f11e37a3640 1 Processor -- start 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.244+0000 7f11e37a3640 1 -- start start 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.244+0000 7f11e37a3640 10 monclient: init 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc142520 con 0x7f11dc1430c0 2024-03-25T19:45:32.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b28a0 con 0x7f11dc0a98f0 2024-03-25T19:45:32.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0b2a30 con 0x7f11dc13f710 2024-03-25T19:45:32.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:32.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e37a3640 10 monclient(hunting): authenticate will time out at 5208.390705s 2024-03-25T19:45:32.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e153f640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f11dc0a98f0 0x7f11dc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:41054/0 (socket says 172.21.15.130:41054) 2024-03-25T19:45:32.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.245+0000 7f11e153f640 1 -- 172.21.15.130:0/4124327254 learned_addr learned my addr 172.21.15.130:0/4124327254 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383943204 0 0) 0x7f11dc142520 con 0x7f11dc1430c0 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 10 monclient(hunting): my global_id is 4398 2024-03-25T19:45:32.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.309+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003520 con 0x7f11dc1430c0 2024-03-25T19:45:32.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 906137485 0 0) 0x7f11c8003520 con 0x7f11dc1430c0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 >> v1:172.21.15.130:6790/0 conn(0x7f11dc13f710 legacy=0x7f11dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 >> v1:172.21.15.163:6789/0 conn(0x7f11dc0a98f0 legacy=0x7f11dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 monclient: found mon.noname-a 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0b44c0 con 0x7f11dc1430c0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: _finish_auth 0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11d4001a10 con 0x7f11dc1430c0 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:32.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11d3fff640 10 monclient: dump: 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11e37a3640 5 monclient: authenticate success, global_id 4398 2024-03-25T19:45:32.333 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.332+0000 7f11e37a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:32.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d4001b70 con 0x7f11dc1430c0 2024-03-25T19:45:32.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:32.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 1 -- 172.21.15.130:0/4124327254 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11d4001f80 con 0x7f11dc1430c0 2024-03-25T19:45:32.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11d3fff640 10 monclient: dump: 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.352+0000 7f11e37a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:32.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 4 set_mon_vals no callback set 2024-03-25T19:45:32.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 10 monclient: shutdown 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 1 -- 172.21.15.130:0/4124327254 >> v1:172.21.15.130:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 1 -- 172.21.15.130:0/4124327254 shutdown_connections 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 1 -- 172.21.15.130:0/4124327254 >> 172.21.15.130:0/4124327254 conn(0x7f11dc0b7ea0 msgr2=0x7f11dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 1 -- 172.21.15.130:0/4124327254 shutdown_connections 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.353+0000 7f11e37a3640 1 -- 172.21.15.130:0/4124327254 wait complete. 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.354+0000 7f11e37a3640 10 monclient: build_initial_monmap 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.354+0000 7f11e37a3640 10 monclient: monmap: 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:32.355 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 librados: starting msgr at 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 librados: starting objecter 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 Processor -- start 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 -- start start 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 librados: setting wanted keys 2024-03-25T19:45:32.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 1 librados: calling monclient init 2024-03-25T19:45:32.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.355+0000 7f11e37a3640 10 monclient: init 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc139a10 con 0x7f11dc0a98f0 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc1d7880 con 0x7f11dc1430c0 2024-03-25T19:45:32.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc137890 con 0x7f11dc13f710 2024-03-25T19:45:32.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:32.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.356+0000 7f11e37a3640 10 monclient(hunting): authenticate will time out at 5208.502023s 2024-03-25T19:45:32.364 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.364+0000 7f11e1d40640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f11dc1430c0 0x7f11dc1e1f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:41096/0 (socket says 172.21.15.130:41096) 2024-03-25T19:45:32.364 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.364+0000 7f11e1d40640 1 -- 172.21.15.130:0/3380150768 learned_addr learned my addr 172.21.15.130:0/3380150768 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:32.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.367+0000 7f11d1ffb640 10 client.?.objecter ms_handle_connect 0x7f11dc1430c0 2024-03-25T19:45:32.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.367+0000 7f11d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.372+0000 7f11d1ffb640 10 client.?.objecter ms_handle_connect 0x7f11dc0a98f0 2024-03-25T19:45:32.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.372+0000 7f11d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.376+0000 7f11d1ffb640 10 client.?.objecter ms_handle_connect 0x7f11dc13f710 2024-03-25T19:45:32.376 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.376+0000 7f11d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:32.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1438137246 0 0) 0x7f11dc139a10 con 0x7f11dc0a98f0 2024-03-25T19:45:32.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 10 monclient(hunting): my global_id is 4401 2024-03-25T19:45:32.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.382+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c0004eb0 con 0x7f11dc0a98f0 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854839188 0 0) 0x7f11dc1d7880 con 0x7f11dc1430c0 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 10 monclient(hunting): my global_id is 4402 2024-03-25T19:45:32.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.390+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc139a10 con 0x7f11dc1430c0 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805788854 0 0) 0x7f11dc137890 con 0x7f11dc13f710 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T19:45:32.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.391+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc1d7880 con 0x7f11dc13f710 2024-03-25T19:45:32.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1301650880 0 0) 0x7f11c0004eb0 con 0x7f11dc0a98f0 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11dc137890 con 0x7f11dc0a98f0 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11d80018a0 con 0x7f11dc0a98f0 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.405 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.405+0000 7f11d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3798781671 0 0) 0x7f11dc139a10 con 0x7f11dc1430c0 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11c0004eb0 con 0x7f11dc1430c0 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11d4001f40 con 0x7f11dc1430c0 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.410+0000 7f11d1ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.410 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.411 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327132578 0 0) 0x7f11dc1d7880 con 0x7f11dc13f710 2024-03-25T19:45:32.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11dc139a10 con 0x7f11dc13f710 2024-03-25T19:45:32.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11cc001860 con 0x7f11dc13f710 2024-03-25T19:45:32.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:32.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.412+0000 7f11d1ffb640 10 monclient(hunting): dump: 2024-03-25T19:45:32.413 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.413 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.413 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.413 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.414 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2624065777 0 0) 0x7f11c0004eb0 con 0x7f11dc1430c0 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 >> v1:172.21.15.130:6790/0 conn(0x7f11dc13f710 legacy=0x7f11dc138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 >> v1:172.21.15.130:6789/0 conn(0x7f11dc0a98f0 legacy=0x7f11dc1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 1 monclient: found mon.b 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc1d84b0 con 0x7f11dc1430c0 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:45:32.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 5 monclient: authenticate success, global_id 4402 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 10 monclient: _renew_subs 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc1d51f0 con 0x7f11dc1430c0 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 10 monclient: _renew_subs 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc138c60 con 0x7f11dc1430c0 2024-03-25T19:45:32.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.422+0000 7f11e37a3640 1 librados: init done 2024-03-25T19:45:32.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.423+0000 7f11e37a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.423+0000 7f11e37a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.423+0000 7f11e37a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:32.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.423+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc1d7a10 con 0x7f11dc1430c0 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11d4003c20 con 0x7f11dc1430c0 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f11d4003fb0 con 0x7f11dc1430c0 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d1ffb640 10 monclient: dump: 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:32.734 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:32.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.734+0000 7f11d37fe640 4 set_mon_vals no callback set 2024-03-25T19:45:32.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.735+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f11d40041d0 con 0x7f11dc1430c0 2024-03-25T19:45:32.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.735+0000 7f11e153f640 10 monclient: get_auth_request con 0x7f11c006e160 auth_method 0 2024-03-25T19:45:32.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.736+0000 7f11d1ffb640 10 client.4402.objecter ms_handle_connect 0x7f11c006e160 2024-03-25T19:45:32.740 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.740+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f11d40020a0 con 0x7f11dc1430c0 2024-03-25T19:45:32.740 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.740+0000 7f11d1ffb640 10 client.4402.objecter ms_dispatch 0x7f11dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:32.740 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.740+0000 7f11d1ffb640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:32.740 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.740+0000 7f11d1ffb640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:32.740 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.740+0000 7f11d1ffb640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T19:45:32.741 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.741+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11d4050670 con 0x7f11dc1430c0 2024-03-25T19:45:32.741 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.741+0000 7f11d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:32.741 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.741+0000 7f11d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.986+0000 7f11e37a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.986+0000 7f11e37a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.986+0000 7f11e37a3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:32.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:32.986+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11dc0627b0 con 0x7f11dc1430c0 2024-03-25T19:45:33.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.027+0000 7f11d1ffb640 1 -- 172.21.15.130:0/3380150768 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f11d40525a0 con 0x7f11dc1430c0 2024-03-25T19:45:33.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.027+0000 7f11d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:33.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.027+0000 7f11d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:33.028 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:33.028 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 10 librados: watch_flush enter 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 10 librados: watch_flush exit 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 10 client.4402.objecter successfully canceled tick 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 >> v1:172.21.15.163:6800/1217380380 conn(0x7f11c006e160 legacy=0x7f11c0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 10 monclient: shutdown 2024-03-25T19:45:33.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 >> v1:172.21.15.163:6789/0 conn(0x7f11dc1430c0 legacy=0x7f11dc1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 shutdown_connections 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 >> 172.21.15.130:0/3380150768 conn(0x7f11dc0b7ea0 msgr2=0x7f11dc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 shutdown_connections 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 -- 172.21.15.130:0/3380150768 wait complete. 2024-03-25T19:45:33.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:33.030+0000 7f11e37a3640 1 librados: shutdown 2024-03-25T19:45:33.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:39.848 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.963+0000 7ffaa40b3640 10 monclient: get_monmap_and_config 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.963+0000 7ffaa40b3640 10 monclient: build_initial_monmap 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.963+0000 7ffaa40b3640 10 monclient: monmap: 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:39.964303+0000 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:39.964303+0000 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:39.963 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:39.964 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:39.964 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.964+0000 7ffaa40b3640 1 Processor -- start 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.964+0000 7ffaa40b3640 1 -- start start 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.964+0000 7ffaa40b3640 10 monclient: init 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c061830 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c06ff10 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c072ae0 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c072f90 con 0x7ffa9c06ff10 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c070bb0 con 0x7ffa9c061830 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0719f0 con 0x7ffa9c072ae0 2024-03-25T19:45:39.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:39.966 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.965+0000 7ffaa40b3640 10 monclient(hunting): authenticate will time out at 5216.110867s 2024-03-25T19:45:39.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.973+0000 7ffaa1e4f640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffa9c061830 0x7ffa9c0723d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:57862/0 (socket says 172.21.15.130:57862) 2024-03-25T19:45:39.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.973+0000 7ffaa1e4f640 1 -- 172.21.15.130:0/1097115364 learned_addr learned my addr 172.21.15.130:0/1097115364 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645341154 0 0) 0x7ffa9c0719f0 con 0x7ffa9c072ae0 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 10 monclient(hunting): _negotiate 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 10 monclient(hunting): my global_id is 4403 2024-03-25T19:45:39.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:39.988+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa88005730 con 0x7ffa9c072ae0 2024-03-25T19:45:40.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.052+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3837096835 0 0) 0x7ffa9c072f90 con 0x7ffa9c06ff10 2024-03-25T19:45:40.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.052+0000 7ffaa0e4d640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.052+0000 7ffaa0e4d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.052+0000 7ffaa0e4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.052+0000 7ffaa0e4d640 10 monclient(hunting): my global_id is 4404 2024-03-25T19:45:40.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.053+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa9c0719f0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.079 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2750968285 0 0) 0x7ffa9c0719f0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 >> v1:172.21.15.130:6790/0 conn(0x7ffa9c072ae0 legacy=0x7ffa9c13c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 >> v1:172.21.15.163:6789/0 conn(0x7ffa9c061830 legacy=0x7ffa9c0723d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 monclient: found mon.noname-a 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c071cd0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa8c001670 con 0x7ffa9c06ff10 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa0e4d640 10 monclient: dump: 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa40b3640 5 monclient: authenticate success, global_id 4404 2024-03-25T19:45:40.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.079+0000 7ffaa40b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:40.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c0017d0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 1 -- 172.21.15.130:0/1097115364 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa8c001ba0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa0e4d640 10 monclient: dump: 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:40.091 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:40.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa40b3640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:40.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.091+0000 7ffaa40b3640 4 set_mon_vals no callback set 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 10 monclient: shutdown 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 1 -- 172.21.15.130:0/1097115364 >> v1:172.21.15.130:6789/0 conn(0x7ffa9c06ff10 legacy=0x7ffa9c13bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 1 -- 172.21.15.130:0/1097115364 shutdown_connections 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 1 -- 172.21.15.130:0/1097115364 >> 172.21.15.130:0/1097115364 conn(0x7ffa9c0b11e0 msgr2=0x7ffa9c0b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 1 -- 172.21.15.130:0/1097115364 shutdown_connections 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.092+0000 7ffaa40b3640 1 -- 172.21.15.130:0/1097115364 wait complete. 2024-03-25T19:45:40.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 10 monclient: build_initial_monmap 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 10 monclient: monmap: 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 librados: starting msgr at 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 librados: starting objecter 2024-03-25T19:45:40.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 Processor -- start 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 -- start start 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 librados: setting wanted keys 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 1 librados: calling monclient init 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.093+0000 7ffaa40b3640 10 monclient: init 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-b con 0x7ffa9c061830 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-a con 0x7ffa9c06ff10 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient(hunting): picked mon.noname-c con 0x7ffa9c072ae0 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c12fdf0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c13a950 con 0x7ffa9c061830 2024-03-25T19:45:40.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa9c0b3ad0 con 0x7ffa9c072ae0 2024-03-25T19:45:40.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:40.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.094+0000 7ffaa40b3640 10 monclient(hunting): authenticate will time out at 5216.239873s 2024-03-25T19:45:40.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.100+0000 7ffaa164e640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ffa9c06ff10 0x7ffa9c12efd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:39194/0 (socket says 172.21.15.130:39194) 2024-03-25T19:45:40.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.100+0000 7ffaa164e640 1 -- 172.21.15.130:0/2887798220 learned_addr learned my addr 172.21.15.130:0/2887798220 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:40.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.103+0000 7ffa927fc640 10 client.?.objecter ms_handle_connect 0x7ffa9c06ff10 2024-03-25T19:45:40.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.103+0000 7ffa927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696698947 0 0) 0x7ffa9c12fdf0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 10 monclient(hunting): my global_id is 4407 2024-03-25T19:45:40.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.113+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa80004eb0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.134+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 870803857 0 0) 0x7ffa80004eb0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.134+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa9c12fdf0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.137+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa8c0035f0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.137+0000 7ffa927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.137+0000 7ffa927fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.137+0000 7ffa927fc640 10 monclient(hunting): dump: 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:40.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.137+0000 7ffa927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:40.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.149+0000 7ffa927fc640 10 client.?.objecter ms_handle_connect 0x7ffa9c061830 2024-03-25T19:45:40.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.149+0000 7ffa927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.153 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.154+0000 7ffa927fc640 10 client.?.objecter ms_handle_connect 0x7ffa9c072ae0 2024-03-25T19:45:40.153 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.154+0000 7ffa927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1954668247 0 0) 0x7ffa9c12fdf0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 >> v1:172.21.15.130:6790/0 conn(0x7ffa9c072ae0 legacy=0x7ffa9c12f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 >> v1:172.21.15.163:6789/0 conn(0x7ffa9c061830 legacy=0x7ffa9c13a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 1 monclient: found mon.a 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa9c0b3db0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffa927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffaa40b3640 5 monclient: authenticate success, global_id 4407 2024-03-25T19:45:40.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffaa40b3640 10 monclient: _renew_subs 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffaa40b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.158+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa9c139d20 con 0x7ffa9c06ff10 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.159+0000 7ffaa40b3640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.159+0000 7ffaa40b3640 10 monclient: _renew_subs 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.159+0000 7ffaa40b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.159+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa9c13ba90 con 0x7ffa9c06ff10 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.159+0000 7ffaa40b3640 1 librados: init done 2024-03-25T19:45:40.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.160+0000 7ffaa40b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.161 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.160+0000 7ffaa40b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.161 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.160+0000 7ffaa40b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.161 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.160+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa9c13cbc0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa8c003b40 con 0x7ffa9c06ff10 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa8c001a40 con 0x7ffa9c06ff10 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa927fc640 10 monclient: dump: 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:40.501 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:40.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.501+0000 7ffa93fff640 4 set_mon_vals no callback set 2024-03-25T19:45:40.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.502+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ffa8c003eb0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.503+0000 7ffaa1e4f640 10 monclient: get_auth_request con 0x7ffa8006dcf0 auth_method 0 2024-03-25T19:45:40.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.503+0000 7ffa927fc640 10 client.4407.objecter ms_handle_connect 0x7ffa8006dcf0 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ffa8c054970 con 0x7ffa9c06ff10 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 10 client.4407.objecter ms_dispatch 0x7ffa9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa8c050480 con 0x7ffa9c06ff10 2024-03-25T19:45:40.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:40.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.510+0000 7ffa927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.761+0000 7ffaa40b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.761+0000 7ffaa40b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.761+0000 7ffaa40b3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:45:40.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.761+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa9c139b10 con 0x7ffa9c06ff10 2024-03-25T19:45:40.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.799+0000 7ffa927fc640 1 -- 172.21.15.130:0/2887798220 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ffa8c0523b0 con 0x7ffa9c06ff10 2024-03-25T19:45:40.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.799+0000 7ffa927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:40.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.799+0000 7ffa927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:40.799 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:40.799 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.801+0000 7ffaa40b3640 10 librados: watch_flush enter 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 10 librados: watch_flush exit 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 10 client.4407.objecter successfully canceled tick 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 >> v1:172.21.15.163:6800/1217380380 conn(0x7ffa8006dcf0 legacy=0x7ffa800701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 10 monclient: shutdown 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 >> v1:172.21.15.130:6789/0 conn(0x7ffa9c06ff10 legacy=0x7ffa9c12efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:40.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 shutdown_connections 2024-03-25T19:45:40.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 >> 172.21.15.130:0/2887798220 conn(0x7ffa9c0b11e0 msgr2=0x7ffa9c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:40.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 shutdown_connections 2024-03-25T19:45:40.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 -- 172.21.15.130:0/2887798220 wait complete. 2024-03-25T19:45:40.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:40.802+0000 7ffaa40b3640 1 librados: shutdown 2024-03-25T19:45:40.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:47.621 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:47.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.736+0000 7f21cbcfb640 10 monclient: get_monmap_and_config 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.736+0000 7f21cbcfb640 10 monclient: build_initial_monmap 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.736+0000 7f21cbcfb640 10 monclient: monmap: 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:47.737147+0000 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:47.737147+0000 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:47.736 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.737+0000 7f21cbcfb640 1 Processor -- start 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.737+0000 7f21cbcfb640 1 -- start start 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.737+0000 7f21cbcfb640 10 monclient: init 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.737+0000 7f21cbcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.737+0000 7f21cbcfb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-a con 0x7f21c40b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-c con 0x7f21c40b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-b con 0x7f21c414be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c40af940 con 0x7f21c40b4d60 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c40b0a70 con 0x7f21c40b1730 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c40b0d50 con 0x7f21c414be90 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:47.738 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.738+0000 7f21cbcfb640 10 monclient(hunting): authenticate will time out at 5223.883877s 2024-03-25T19:45:47.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.748+0000 7f21c9a97640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f21c40b4d60 0x7f21c413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:47962/0 (socket says 172.21.15.130:47962) 2024-03-25T19:45:47.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.748+0000 7f21c9a97640 1 -- 172.21.15.130:0/3369197880 learned_addr learned my addr 172.21.15.130:0/3369197880 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1166787830 0 0) 0x7f21c40b0d50 con 0x7f21c414be90 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 10 monclient(hunting): my global_id is 4406 2024-03-25T19:45:47.757 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.758+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21b0003520 con 0x7f21c414be90 2024-03-25T19:45:47.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 234800949 0 0) 0x7f21c40b0a70 con 0x7f21c40b1730 2024-03-25T19:45:47.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.761 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 10 monclient(hunting): my global_id is 4408 2024-03-25T19:45:47.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.762+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c40b0d50 con 0x7f21c40b1730 2024-03-25T19:45:47.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2006479673 0 0) 0x7f21b0003520 con 0x7f21c414be90 2024-03-25T19:45:47.764 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 >> v1:172.21.15.163:6789/0 conn(0x7f21c40b1730 legacy=0x7f21c40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 >> v1:172.21.15.130:6789/0 conn(0x7f21c40b4d60 legacy=0x7f21c413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: _finish_hunting 0 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 monclient: found mon.noname-b 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c40b1030 con 0x7f21c414be90 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: _finish_auth 0 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: _check_auth_tickets 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f21bc001a10 con 0x7f21c414be90 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21cbcfb640 5 monclient: authenticate success, global_id 4406 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.764+0000 7f21c8a95640 10 monclient: dump: 2024-03-25T19:45:47.765 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.765+0000 7f21cbcfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:47.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21bc001b70 con 0x7f21c414be90 2024-03-25T19:45:47.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 1 -- 172.21.15.130:0/3369197880 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f21bc001f80 con 0x7f21c414be90 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21cbcfb640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21cbcfb640 4 set_mon_vals no callback set 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.773+0000 7f21c8a95640 10 monclient: dump: 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:47.773 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:47.774 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:47.774 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 10 monclient: shutdown 2024-03-25T19:45:47.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:47.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 1 -- 172.21.15.130:0/3369197880 >> v1:172.21.15.130:6790/0 conn(0x7f21c414be90 legacy=0x7f21c414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 1 -- 172.21.15.130:0/3369197880 shutdown_connections 2024-03-25T19:45:47.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 1 -- 172.21.15.130:0/3369197880 >> 172.21.15.130:0/3369197880 conn(0x7f21c40617a0 msgr2=0x7f21c4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 1 -- 172.21.15.130:0/3369197880 shutdown_connections 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.774+0000 7f21cbcfb640 1 -- 172.21.15.130:0/3369197880 wait complete. 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 10 monclient: build_initial_monmap 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 10 monclient: monmap: 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:47.776 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 librados: starting msgr at 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 librados: starting objecter 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 Processor -- start 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 -- start start 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 librados: setting wanted keys 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 1 librados: calling monclient init 2024-03-25T19:45:47.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.775+0000 7f21cbcfb640 10 monclient: init 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-c con 0x7f21c40b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-b con 0x7f21c40b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient(hunting): picked mon.noname-a con 0x7f21c414be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:47.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4141d30 con 0x7f21c414be90 2024-03-25T19:45:47.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4140960 con 0x7f21c40b4d60 2024-03-25T19:45:47.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21c4137fd0 con 0x7f21c40b1730 2024-03-25T19:45:47.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:47.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.776+0000 7f21cbcfb640 10 monclient(hunting): authenticate will time out at 5223.922321s 2024-03-25T19:45:47.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.788+0000 7f21c9296640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f21c40b4d60 0x7f21c4142430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:43530/0 (socket says 172.21.15.130:43530) 2024-03-25T19:45:47.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.788+0000 7f21c9296640 1 -- 172.21.15.130:0/921979965 learned_addr learned my addr 172.21.15.130:0/921979965 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.794+0000 7f21ba7fc640 10 client.?.objecter ms_handle_connect 0x7f21c40b4d60 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.794+0000 7f21ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.794+0000 7f21ba7fc640 10 client.?.objecter ms_handle_connect 0x7f21c414be90 2024-03-25T19:45:47.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.794+0000 7f21ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.795+0000 7f21ba7fc640 10 client.?.objecter ms_handle_connect 0x7f21c40b1730 2024-03-25T19:45:47.795 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.795+0000 7f21ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179368303 0 0) 0x7f21c4140960 con 0x7f21c40b4d60 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 10 monclient(hunting): my global_id is 4411 2024-03-25T19:45:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.806+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21a8004eb0 con 0x7f21c40b4d60 2024-03-25T19:45:47.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519300448 0 0) 0x7f21c4141d30 con 0x7f21c414be90 2024-03-25T19:45:47.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c4140960 con 0x7f21c414be90 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706444892 0 0) 0x7f21c4137fd0 con 0x7f21c40b1730 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 10 monclient(hunting): my global_id is 4409 2024-03-25T19:45:47.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.812+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21c4141d30 con 0x7f21c40b1730 2024-03-25T19:45:47.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278707027 0 0) 0x7f21a8004eb0 con 0x7f21c40b4d60 2024-03-25T19:45:47.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21c4137fd0 con 0x7f21c40b4d60 2024-03-25T19:45:47.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f21b4001a10 con 0x7f21c40b4d60 2024-03-25T19:45:47.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.817+0000 7f21ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.829+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1485694048 0 0) 0x7f21c4140960 con 0x7f21c414be90 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21a8004eb0 con 0x7f21c414be90 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f21bc001e80 con 0x7f21c414be90 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient(hunting): dump: 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:47.831 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1722869303 0 0) 0x7f21c4137fd0 con 0x7f21c40b4d60 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 >> v1:172.21.15.130:6790/0 conn(0x7f21c40b1730 legacy=0x7f21c4131cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 >> v1:172.21.15.130:6789/0 conn(0x7f21c414be90 legacy=0x7f21c41e2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 monclient: found mon.b 2024-03-25T19:45:47.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21c4138160 con 0x7f21c40b4d60 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient: _finish_auth 0 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 5 monclient: authenticate success, global_id 4411 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 10 monclient: _renew_subs 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21c4140380 con 0x7f21c40b4d60 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 10 client.4411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 10 monclient: _renew_subs 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21c4145590 con 0x7f21c40b4d60 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.830+0000 7f21cbcfb640 1 librados: init done 2024-03-25T19:45:47.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.831+0000 7f21cbcfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:47.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.831+0000 7f21cbcfb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:47.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.831+0000 7f21cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:47.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:47.832+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21c40627b0 con 0x7f21c40b4d60 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21b4001c50 con 0x7f21c40b4d60 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f21b4003a60 con 0x7f21c40b4d60 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:48.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21ba7fc640 10 monclient: dump: 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:48.117 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.116+0000 7f21bbfff640 4 set_mon_vals no callback set 2024-03-25T19:45:48.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.117+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f21b4003c80 con 0x7f21c40b4d60 2024-03-25T19:45:48.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.118+0000 7f21c9a97640 10 monclient: get_auth_request con 0x7f21a806df70 auth_method 0 2024-03-25T19:45:48.118 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.118+0000 7f21ba7fc640 10 client.4411.objecter ms_handle_connect 0x7f21a806df70 2024-03-25T19:45:48.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.122+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f21b4054830 con 0x7f21c40b4d60 2024-03-25T19:45:48.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.122+0000 7f21ba7fc640 10 client.4411.objecter ms_dispatch 0x7f21c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:48.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.122+0000 7f21ba7fc640 3 client.4411.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:48.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.122+0000 7f21ba7fc640 3 client.4411.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:48.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.122+0000 7f21ba7fc640 20 client.4411.objecter dump_active .. 0 homeless 2024-03-25T19:45:48.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.126+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f21b4050340 con 0x7f21c40b4d60 2024-03-25T19:45:48.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.126+0000 7f21ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:48.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.126+0000 7f21ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:48.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.343+0000 7f21cbcfb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.343+0000 7f21cbcfb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.343+0000 7f21cbcfb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:48.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.343+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21c4140720 con 0x7f21c40b4d60 2024-03-25T19:45:48.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.368+0000 7f21ba7fc640 1 -- 172.21.15.130:0/921979965 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f21b4052270 con 0x7f21c40b4d60 2024-03-25T19:45:48.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.368+0000 7f21ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:48.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.368+0000 7f21ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:48.368 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:48.368 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:48.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 10 librados: watch_flush enter 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 10 librados: watch_flush exit 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 20 client.4411.objecter shutdown clearing up homeless session... 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 10 client.4411.objecter successfully canceled tick 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 >> v1:172.21.15.163:6800/1217380380 conn(0x7f21a806df70 legacy=0x7f21a8070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 10 monclient: shutdown 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 >> v1:172.21.15.163:6789/0 conn(0x7f21c40b4d60 legacy=0x7f21c4142430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 shutdown_connections 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 >> 172.21.15.130:0/921979965 conn(0x7f21c40617a0 msgr2=0x7f21c40b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.370+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 shutdown_connections 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.371+0000 7f21cbcfb640 1 -- 172.21.15.130:0/921979965 wait complete. 2024-03-25T19:45:48.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:48.371+0000 7f21cbcfb640 1 librados: shutdown 2024-03-25T19:45:48.382 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:45:55.189 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:45:55.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.305+0000 7f86beca8640 10 monclient: get_monmap_and_config 2024-03-25T19:45:55.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.305+0000 7f86beca8640 10 monclient: build_initial_monmap 2024-03-25T19:45:55.305 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.305+0000 7f86beca8640 10 monclient: monmap: 2024-03-25T19:45:55.305 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:55.305 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:45:55.306633+0000 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:45:55.306633+0000 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:45:55.306 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.306+0000 7f86beca8640 1 Processor -- start 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.306+0000 7f86beca8640 1 -- start start 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.306+0000 7f86beca8640 10 monclient: init 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-b con 0x7f86b80b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:55.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-a con 0x7f86b80b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-c con 0x7f86b814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b80af940 con 0x7f86b80b1730 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b80b0a70 con 0x7f86b814be90 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b80b0d50 con 0x7f86b80b4d60 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:55.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.307+0000 7f86beca8640 10 monclient(hunting): authenticate will time out at 5231.453062s 2024-03-25T19:45:55.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.318+0000 7f86bd245640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f86b814be90 0x7f86b814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:48232/0 (socket says 172.21.15.130:48232) 2024-03-25T19:45:55.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.318+0000 7f86bd245640 1 -- 172.21.15.130:0/4046662798 learned_addr learned my addr 172.21.15.130:0/4046662798 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:55.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134528914 0 0) 0x7f86b80b0a70 con 0x7f86b814be90 2024-03-25T19:45:55.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T19:45:55.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.335+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86a8003520 con 0x7f86b814be90 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3840134266 0 0) 0x7f86b80af940 con 0x7f86b80b1730 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 10 monclient(hunting): my global_id is 4416 2024-03-25T19:45:55.337 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.337+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b80b0a70 con 0x7f86b80b1730 2024-03-25T19:45:55.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808390361 0 0) 0x7f86b80b0d50 con 0x7f86b80b4d60 2024-03-25T19:45:55.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 10 monclient(hunting): my global_id is 4412 2024-03-25T19:45:55.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.339+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b80af940 con 0x7f86b80b4d60 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2059831146 0 0) 0x7f86b80b0a70 con 0x7f86b80b1730 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 >> v1:172.21.15.130:6790/0 conn(0x7f86b80b4d60 legacy=0x7f86b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 >> v1:172.21.15.163:6789/0 conn(0x7f86b814be90 legacy=0x7f86b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: _finish_hunting 0 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 monclient: found mon.noname-a 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b80b1030 con 0x7f86b80b1730 2024-03-25T19:45:55.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: _finish_auth 0 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: _check_auth_tickets 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86a4001670 con 0x7f86b80b1730 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86beca8640 5 monclient: authenticate success, global_id 4416 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86beca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86b77fe640 10 monclient: dump: 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.350 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.351 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.351 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.349+0000 7f86beca8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86a40017d0 con 0x7f86b80b1730 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 1 -- 172.21.15.130:0/4046662798 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86a4001be0 con 0x7f86b80b1730 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86b77fe640 10 monclient: dump: 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.366 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86beca8640 10 monclient: get_monmap_and_config success 2024-03-25T19:45:55.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.366+0000 7f86beca8640 4 set_mon_vals no callback set 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 10 monclient: shutdown 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 1 -- 172.21.15.130:0/4046662798 >> v1:172.21.15.130:6789/0 conn(0x7f86b80b1730 legacy=0x7f86b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 1 -- 172.21.15.130:0/4046662798 shutdown_connections 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 1 -- 172.21.15.130:0/4046662798 >> 172.21.15.130:0/4046662798 conn(0x7f86b80617a0 msgr2=0x7f86b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:55.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 1 -- 172.21.15.130:0/4046662798 shutdown_connections 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 1 -- 172.21.15.130:0/4046662798 wait complete. 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 10 monclient: build_initial_monmap 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.367+0000 7f86beca8640 10 monclient: monmap: 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:45:55.370 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 librados: starting msgr at 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 librados: starting objecter 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 Processor -- start 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 -- start start 2024-03-25T19:45:55.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 librados: setting wanted keys 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 1 librados: calling monclient init 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.368+0000 7f86beca8640 10 monclient: init 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient: _reopen_session rank -1 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-b con 0x7f86b80b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-a con 0x7f86b80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient(hunting): picked mon.noname-c con 0x7f86b814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8143d40 con 0x7f86b80b4d60 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8138480 con 0x7f86b80b1730 2024-03-25T19:45:55.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86b8141130 con 0x7f86b814be90 2024-03-25T19:45:55.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient(hunting): _renew_subs 2024-03-25T19:45:55.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.369+0000 7f86beca8640 10 monclient(hunting): authenticate will time out at 5231.515250s 2024-03-25T19:45:55.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.375+0000 7f86bca44640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f86b80b1730 0x7f86b8137c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:48258/0 (socket says 172.21.15.130:48258) 2024-03-25T19:45:55.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.375+0000 7f86bca44640 1 -- 172.21.15.130:0/2213258282 learned_addr learned my addr 172.21.15.130:0/2213258282 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:45:55.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.379+0000 7f86b57fa640 10 client.?.objecter ms_handle_connect 0x7f86b80b1730 2024-03-25T19:45:55.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.379+0000 7f86b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.384+0000 7f86b57fa640 10 client.?.objecter ms_handle_connect 0x7f86b80b4d60 2024-03-25T19:45:55.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.384+0000 7f86b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.385+0000 7f86b57fa640 10 client.?.objecter ms_handle_connect 0x7f86b814be90 2024-03-25T19:45:55.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.385+0000 7f86b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2056309009 0 0) 0x7f86b8141130 con 0x7f86b814be90 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 10 monclient(hunting): my global_id is 4415 2024-03-25T19:45:55.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.392+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f869c004eb0 con 0x7f86b814be90 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500777416 0 0) 0x7f86b8138480 con 0x7f86b80b1730 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 10 monclient(hunting): my global_id is 4417 2024-03-25T19:45:55.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.396+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8141130 con 0x7f86b80b1730 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4267707449 0 0) 0x7f86b8143d40 con 0x7f86b80b4d60 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 10 monclient(hunting): _negotiate 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 10 monclient(hunting): my global_id is 4419 2024-03-25T19:45:55.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.398+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b8138480 con 0x7f86b80b4d60 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 801009045 0 0) 0x7f869c004eb0 con 0x7f86b814be90 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86b8143d40 con 0x7f86b814be90 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86b0001af0 con 0x7f86b814be90 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): dump: 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.404 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3941359747 0 0) 0x7f86b8141130 con 0x7f86b80b1730 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f869c004eb0 con 0x7f86b80b1730 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86ac0019f0 con 0x7f86b80b1730 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.404+0000 7f86b57fa640 10 monclient(hunting): dump: 2024-03-25T19:45:55.405 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.406 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3896200286 0 0) 0x7f86b8138480 con 0x7f86b80b4d60 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86b8141130 con 0x7f86b80b4d60 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86a4001ae0 con 0x7f86b80b4d60 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.409+0000 7f86b57fa640 10 monclient(hunting): dump: 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.409 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.410 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3930945037 0 0) 0x7f869c004eb0 con 0x7f86b80b1730 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 >> v1:172.21.15.130:6790/0 conn(0x7f86b814be90 legacy=0x7f86b81d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 >> v1:172.21.15.130:6789/0 conn(0x7f86b80b4d60 legacy=0x7f86b8142890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 10 monclient: _finish_hunting 0 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 1 monclient: found mon.b 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b8137660 con 0x7f86b80b1730 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 10 monclient: _finish_auth 0 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 10 monclient: _check_auth_tickets 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:45:55.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 5 monclient: authenticate success, global_id 4417 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 10 monclient: _renew_subs 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86b8137360 con 0x7f86b80b1730 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 10 client.4417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 10 monclient: _renew_subs 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.412+0000 7f86beca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.413+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86b8143180 con 0x7f86b80b1730 2024-03-25T19:45:55.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.413+0000 7f86beca8640 1 librados: init done 2024-03-25T19:45:55.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.414+0000 7f86beca8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.414+0000 7f86beca8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.414+0000 7f86beca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:55.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.414+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86b8140f30 con 0x7f86b80b1730 2024-03-25T19:45:55.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f86ac003610 con 0x7f86b80b1730 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f86ac0039a0 con 0x7f86b80b1730 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b57fa640 10 monclient: dump: 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:45:55.693 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.693+0000 7f86b6ffd640 4 set_mon_vals no callback set 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.694+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f86ac003bc0 con 0x7f86b80b1730 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.694+0000 7f86b7fff640 10 monclient: get_auth_request con 0x7f869c06e160 auth_method 0 2024-03-25T19:45:55.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.695+0000 7f86b57fa640 10 client.4417.objecter ms_handle_connect 0x7f869c06e160 2024-03-25T19:45:55.699 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.699+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f86ac054800 con 0x7f86b80b1730 2024-03-25T19:45:55.699 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.699+0000 7f86b57fa640 10 client.4417.objecter ms_dispatch 0x7f86b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:45:55.699 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.699+0000 7f86b57fa640 3 client.4417.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:45:55.699 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.699+0000 7f86b57fa640 3 client.4417.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:45:55.699 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.699+0000 7f86b57fa640 20 client.4417.objecter dump_active .. 0 homeless 2024-03-25T19:45:55.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.700+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f86ac050310 con 0x7f86b80b1730 2024-03-25T19:45:55.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.700+0000 7f86b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:45:55.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.700+0000 7f86b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:45:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.949+0000 7f86beca8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.949+0000 7f86beca8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.949+0000 7f86beca8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:45:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:55.949+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86b80627b0 con 0x7f86b80b1730 2024-03-25T19:45:56.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.122+0000 7f86b57fa640 1 -- 172.21.15.130:0/2213258282 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f86ac052240 con 0x7f86b80b1730 2024-03-25T19:45:56.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.122+0000 7f86b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:45:56.122 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.122+0000 7f86b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:45:56.122 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:45:56.122 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:45:56.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 10 librados: watch_flush enter 2024-03-25T19:45:56.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 10 librados: watch_flush exit 2024-03-25T19:45:56.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 20 client.4417.objecter shutdown clearing up homeless session... 2024-03-25T19:45:56.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 10 client.4417.objecter successfully canceled tick 2024-03-25T19:45:56.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 >> v1:172.21.15.163:6800/1217380380 conn(0x7f869c06e160 legacy=0x7f869c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 10 monclient: shutdown 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 >> v1:172.21.15.163:6789/0 conn(0x7f86b80b1730 legacy=0x7f86b8137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 shutdown_connections 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 >> 172.21.15.130:0/2213258282 conn(0x7f86b80617a0 msgr2=0x7f86b80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 shutdown_connections 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 -- 172.21.15.130:0/2213258282 wait complete. 2024-03-25T19:45:56.125 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:45:56.124+0000 7f86beca8640 1 librados: shutdown 2024-03-25T19:45:56.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:00.105 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:45.038 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.111 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:45.044 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.168 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:04:45.101 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.169 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:45.070 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.179 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:45.080 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.185 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:45.118 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.194 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:45.127 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.215 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:04:45.116 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:00.230 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:04:45.163 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:02.943 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:03.060 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.060+0000 7fa602625640 10 monclient: get_monmap_and_config 2024-03-25T19:46:03.060 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.060+0000 7fa602625640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.060 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.060+0000 7fa602625640 10 monclient: monmap: 2024-03-25T19:46:03.060 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:03.061743+0000 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:03.061743+0000 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:03.061 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.061+0000 7fa602625640 1 Processor -- start 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 1 -- start start 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient: init 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc142520 con 0x7fa5fc1430c0 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0b28a0 con 0x7fa5fc13f710 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc0b2a30 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.062+0000 7fa602625640 10 monclient(hunting): authenticate will time out at 5239.208364s 2024-03-25T19:46:03.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.069+0000 7fa5fbfff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fa5fc0a98f0 0x7fa5fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:46988/0 (socket says 172.21.15.130:46988) 2024-03-25T19:46:03.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.069+0000 7fa5fbfff640 1 -- 172.21.15.130:0/2842786594 learned_addr learned my addr 172.21.15.130:0/2842786594 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984779455 0 0) 0x7fa5fc0b2a30 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 10 monclient(hunting): my global_id is 4418 2024-03-25T19:46:03.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.078+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5e8003520 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660675192 0 0) 0x7fa5fc142520 con 0x7fa5fc1430c0 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 10 monclient(hunting): my global_id is 4422 2024-03-25T19:46:03.081 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.081+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc0b2a30 con 0x7fa5fc1430c0 2024-03-25T19:46:03.083 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.083+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3905789887 0 0) 0x7fa5e8003520 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 >> v1:172.21.15.163:6789/0 conn(0x7fa5fc13f710 legacy=0x7fa5fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 >> v1:172.21.15.130:6789/0 conn(0x7fa5fc1430c0 legacy=0x7fa5fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 1 monclient: found mon.noname-b 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc0b44c0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5ec001690 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa602625640 5 monclient: authenticate success, global_id 4418 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa602625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.084 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa5fb7fe640 10 monclient: dump: 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.085 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.084+0000 7fa602625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:03.097 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ec0017f0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 1 -- 172.21.15.130:0/2842786594 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5ec001c00 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa602625640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa602625640 4 set_mon_vals no callback set 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.098+0000 7fa5fb7fe640 10 monclient: dump: 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.098 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.099 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.099 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.099 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.099 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 10 monclient: shutdown 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 1 -- 172.21.15.130:0/2842786594 >> v1:172.21.15.130:6790/0 conn(0x7fa5fc0a98f0 legacy=0x7fa5fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 1 -- 172.21.15.130:0/2842786594 shutdown_connections 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 1 -- 172.21.15.130:0/2842786594 >> 172.21.15.130:0/2842786594 conn(0x7fa5fc0b7ea0 msgr2=0x7fa5fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 1 -- 172.21.15.130:0/2842786594 shutdown_connections 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 1 -- 172.21.15.130:0/2842786594 wait complete. 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 10 monclient: build_initial_monmap 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.099+0000 7fa602625640 10 monclient: monmap: 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:03.101 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 librados: starting msgr at 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 librados: starting objecter 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 Processor -- start 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 -- start start 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 librados: setting wanted keys 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 1 librados: calling monclient init 2024-03-25T19:46:03.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.100+0000 7fa602625640 10 monclient: init 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient(hunting): picked mon.noname-c con 0x7fa5fc0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient(hunting): picked mon.noname-a con 0x7fa5fc13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient(hunting): picked mon.noname-b con 0x7fa5fc1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc1370a0 con 0x7fa5fc13f710 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc135090 con 0x7fa5fc1430c0 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5fc135220 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:03.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.101+0000 7fa602625640 10 monclient(hunting): authenticate will time out at 5239.246826s 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.109+0000 7fa600bc2640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa5fc1430c0 0x7fa5fc1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:43320/0 (socket says 172.21.15.130:43320) 2024-03-25T19:46:03.108 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.109+0000 7fa600bc2640 1 -- 172.21.15.130:0/2103315437 learned_addr learned my addr 172.21.15.130:0/2103315437 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:03.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.110+0000 7fa5f97fa640 10 client.?.objecter ms_handle_connect 0x7fa5fc1430c0 2024-03-25T19:46:03.109 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.110+0000 7fa5f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.116+0000 7fa5f97fa640 10 client.?.objecter ms_handle_connect 0x7fa5fc13f710 2024-03-25T19:46:03.116 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.116+0000 7fa5f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.117 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.118+0000 7fa5f97fa640 10 client.?.objecter ms_handle_connect 0x7fa5fc0a98f0 2024-03-25T19:46:03.117 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.118+0000 7fa5f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995443702 0 0) 0x7fa5fc135220 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 10 monclient(hunting): my global_id is 4421 2024-03-25T19:46:03.123 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.123+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5dc004eb0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.124+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190034589 0 0) 0x7fa5fc135090 con 0x7fa5fc1430c0 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.125+0000 7fa5f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.125+0000 7fa5f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.125+0000 7fa5f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.125+0000 7fa5f97fa640 10 monclient(hunting): my global_id is 4423 2024-03-25T19:46:03.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.125+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc135220 con 0x7fa5fc1430c0 2024-03-25T19:46:03.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3579265877 0 0) 0x7fa5dc004eb0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc135090 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5ec001bc0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 10 monclient(hunting): dump: 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.137+0000 7fa5f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210780203 0 0) 0x7fa5fc1370a0 con 0x7fa5fc13f710 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 10 monclient(hunting): my global_id is 4425 2024-03-25T19:46:03.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.138+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5dc004eb0 con 0x7fa5fc13f710 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2464266990 0 0) 0x7fa5fc135220 con 0x7fa5fc1430c0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc1370a0 con 0x7fa5fc1430c0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5f4001c10 con 0x7fa5fc1430c0 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:03.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.144+0000 7fa5f97fa640 10 monclient(hunting): dump: 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.144 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 413129415 0 0) 0x7fa5dc004eb0 con 0x7fa5fc13f710 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa5fc135220 con 0x7fa5fc13f710 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5e4001860 con 0x7fa5fc13f710 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.152+0000 7fa5f97fa640 10 monclient(hunting): dump: 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.152 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.153 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.153 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.153 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.153 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2582912261 0 0) 0x7fa5fc135090 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 >> v1:172.21.15.163:6789/0 conn(0x7fa5fc1430c0 legacy=0x7fa5fc1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 >> v1:172.21.15.130:6789/0 conn(0x7fa5fc13f710 legacy=0x7fa5fc1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 1 monclient: found mon.c 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc1d5000 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa5f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 5 monclient: authenticate success, global_id 4421 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 10 monclient: _renew_subs 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc1d52b0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 10 monclient: _renew_subs 2024-03-25T19:46:03.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc135910 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.156+0000 7fa602625640 1 librados: init done 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.158+0000 7fa602625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.158+0000 7fa602625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.158+0000 7fa602625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.158+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc0627b0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5ec004a90 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa5ec001250 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5f97fa640 10 monclient: dump: 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:03.460 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.460+0000 7fa5faffd640 4 set_mon_vals no callback set 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.461+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fa5ec01ac50 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.462+0000 7fa5f3fff640 10 monclient: get_auth_request con 0x7fa5dc06e110 auth_method 0 2024-03-25T19:46:03.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.462+0000 7fa5f97fa640 10 client.4421.objecter ms_handle_connect 0x7fa5dc06e110 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fa5ec054860 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 10 client.4421.objecter ms_dispatch 0x7fa5fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 3 client.4421.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 3 client.4421.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 20 client.4421.objecter dump_active .. 0 homeless 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5ec0501a0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:03.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.466+0000 7fa5f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:03.706 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.706+0000 7fa602625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.706+0000 7fa602625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.706+0000 7fa602625640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:03.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.706+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5fc13bfe0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.726 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.727+0000 7fa5f97fa640 1 -- 172.21.15.130:0/2103315437 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fa5ec0522a0 con 0x7fa5fc0a98f0 2024-03-25T19:46:03.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.727+0000 7fa5f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:03.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.727+0000 7fa5f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:03.727 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:03.727 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 10 librados: watch_flush enter 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 10 librados: watch_flush exit 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 20 client.4421.objecter shutdown clearing up homeless session... 2024-03-25T19:46:03.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 10 client.4421.objecter successfully canceled tick 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 >> v1:172.21.15.163:6800/1217380380 conn(0x7fa5dc06e110 legacy=0x7fa5dc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 10 monclient: shutdown 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 >> v1:172.21.15.130:6790/0 conn(0x7fa5fc0a98f0 legacy=0x7fa5fc1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 shutdown_connections 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 >> 172.21.15.130:0/2103315437 conn(0x7fa5fc0b7ea0 msgr2=0x7fa5fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 shutdown_connections 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 -- 172.21.15.130:0/2103315437 wait complete. 2024-03-25T19:46:03.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:03.729+0000 7fa602625640 1 librados: shutdown 2024-03-25T19:46:03.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:10.548 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:10.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.663+0000 7f4eb6bc8640 10 monclient: get_monmap_and_config 2024-03-25T19:46:10.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.663+0000 7f4eb6bc8640 10 monclient: build_initial_monmap 2024-03-25T19:46:10.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.663+0000 7f4eb6bc8640 10 monclient: monmap: 2024-03-25T19:46:10.664 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:10.665577+0000 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:10.665577+0000 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:10.665 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.664+0000 7f4eb6bc8640 1 Processor -- start 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.664+0000 7f4eb6bc8640 1 -- start start 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.664+0000 7f4eb6bc8640 10 monclient: init 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:10.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-c con 0x7f4eb00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-a con 0x7f4eb013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-b con 0x7f4eb01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb0142520 con 0x7f4eb013f710 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00b28a0 con 0x7f4eb00a98f0 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00b2a30 con 0x7f4eb01430c0 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:10.667 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.665+0000 7f4eb6bc8640 10 monclient(hunting): authenticate will time out at 5246.812368s 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.674+0000 7f4eb4964640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4eb00a98f0 0x7f4eb00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51730/0 (socket says 172.21.15.130:51730) 2024-03-25T19:46:10.674 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.674+0000 7f4eb4964640 1 -- 172.21.15.130:0/606269720 learned_addr learned my addr 172.21.15.130:0/606269720 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:10.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880849227 0 0) 0x7f4eb0142520 con 0x7f4eb013f710 2024-03-25T19:46:10.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 10 monclient(hunting): my global_id is 4428 2024-03-25T19:46:10.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.692+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea0003520 con 0x7f4eb013f710 2024-03-25T19:46:10.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934818768 0 0) 0x7f4eb00b28a0 con 0x7f4eb00a98f0 2024-03-25T19:46:10.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 10 monclient(hunting): my global_id is 4426 2024-03-25T19:46:10.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.702+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb0142520 con 0x7f4eb00a98f0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3165348952 0 0) 0x7f4ea0003520 con 0x7f4eb013f710 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 >> v1:172.21.15.130:6790/0 conn(0x7f4eb01430c0 legacy=0x7f4eb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 >> v1:172.21.15.163:6789/0 conn(0x7f4eb00a98f0 legacy=0x7f4eb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 monclient: found mon.noname-a 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb00b44c0 con 0x7f4eb013f710 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:10.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4e9c001690 con 0x7f4eb013f710 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eaf7fe640 10 monclient: dump: 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:10.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eb6bc8640 5 monclient: authenticate success, global_id 4428 2024-03-25T19:46:10.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.713+0000 7f4eb6bc8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e9c0017f0 con 0x7f4eb013f710 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 1 -- 172.21.15.130:0/606269720 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4e9c001c00 con 0x7f4eb013f710 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eb6bc8640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eb6bc8640 4 set_mon_vals no callback set 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.728+0000 7f4eaf7fe640 10 monclient: dump: 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.729 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:10.730 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:10.730 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:10.730 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:10.730 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:10.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 10 monclient: shutdown 2024-03-25T19:46:10.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/606269720 >> v1:172.21.15.130:6789/0 conn(0x7f4eb013f710 legacy=0x7f4eb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/606269720 shutdown_connections 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/606269720 >> 172.21.15.130:0/606269720 conn(0x7f4eb00b7ea0 msgr2=0x7f4eb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/606269720 shutdown_connections 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/606269720 wait complete. 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 10 monclient: build_initial_monmap 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.729+0000 7f4eb6bc8640 10 monclient: monmap: 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:10.732 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 librados: starting msgr at 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 librados: starting objecter 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 Processor -- start 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 -- start start 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 librados: setting wanted keys 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 1 librados: calling monclient init 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.730+0000 7f4eb6bc8640 10 monclient: init 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-c con 0x7f4eb00a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-a con 0x7f4eb013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient(hunting): picked mon.noname-b con 0x7f4eb01430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00abd70 con 0x7f4eb013f710 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb01d7710 con 0x7f4eb01430c0 2024-03-25T19:46:10.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb01d8340 con 0x7f4eb00a98f0 2024-03-25T19:46:10.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:10.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.731+0000 7f4eb6bc8640 10 monclient(hunting): authenticate will time out at 5246.877707s 2024-03-25T19:46:10.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.748+0000 7f4eb5165640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f4eb01430c0 0x7f4eb01d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51788/0 (socket says 172.21.15.130:51788) 2024-03-25T19:46:10.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.748+0000 7f4eb5165640 1 -- 172.21.15.130:0/544214339 learned_addr learned my addr 172.21.15.130:0/544214339 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:10.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.751+0000 7f4ead7fa640 10 client.?.objecter ms_handle_connect 0x7f4eb01430c0 2024-03-25T19:46:10.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.751+0000 7f4ead7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.753+0000 7f4ead7fa640 10 client.?.objecter ms_handle_connect 0x7f4eb013f710 2024-03-25T19:46:10.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.753+0000 7f4ead7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275741364 0 0) 0x7f4eb01d7710 con 0x7f4eb01430c0 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T19:46:10.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.757+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e94004eb0 con 0x7f4eb01430c0 2024-03-25T19:46:10.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2168846697 0 0) 0x7f4eb00abd70 con 0x7f4eb013f710 2024-03-25T19:46:10.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:10.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:10.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:10.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 10 monclient(hunting): my global_id is 4431 2024-03-25T19:46:10.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.764+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4eb01d7710 con 0x7f4eb013f710 2024-03-25T19:46:10.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.765+0000 7f4ead7fa640 10 client.?.objecter ms_handle_connect 0x7f4eb00a98f0 2024-03-25T19:46:10.766 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.765+0000 7f4ead7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183275309 0 0) 0x7f4e94004eb0 con 0x7f4eb01430c0 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4eb00abd70 con 0x7f4eb01430c0 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4ea8001c90 con 0x7f4eb01430c0 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 10 monclient(hunting): dump: 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:10.770 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:10.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.769+0000 7f4ead7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:10.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 710645781 0 0) 0x7f4eb00abd70 con 0x7f4eb01430c0 2024-03-25T19:46:10.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 >> v1:172.21.15.130:6790/0 conn(0x7f4eb00a98f0 legacy=0x7f4eb0137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 >> v1:172.21.15.130:6789/0 conn(0x7f4eb013f710 legacy=0x7f4eb01d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 1 monclient: found mon.b 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb01d84d0 con 0x7f4eb01430c0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4ead7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4eb6bc8640 5 monclient: authenticate success, global_id 4429 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4eb6bc8640 10 monclient: _renew_subs 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4eb6bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.775+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb01d41e0 con 0x7f4eb01430c0 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 monclient: _renew_subs 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb01d5150 con 0x7f4eb01430c0 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 1 librados: init done 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:10.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:46:10.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:10.776+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb00627b0 con 0x7f4eb01430c0 2024-03-25T19:46:11.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4ea8003990 con 0x7f4eb01430c0 2024-03-25T19:46:11.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:11.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4ea8003ce0 con 0x7f4eb01430c0 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4ead7fa640 10 monclient: dump: 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:11.066 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:11.067 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:11.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.064+0000 7f4eaeffd640 4 set_mon_vals no callback set 2024-03-25T19:46:11.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.065+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f4ea8003f00 con 0x7f4eb01430c0 2024-03-25T19:46:11.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.066+0000 7f4eb4964640 10 monclient: get_auth_request con 0x7f4e9406dd40 auth_method 0 2024-03-25T19:46:11.067 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.066+0000 7f4ead7fa640 10 client.4429.objecter ms_handle_connect 0x7f4e9406dd40 2024-03-25T19:46:11.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.071+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f4ea8054ad0 con 0x7f4eb01430c0 2024-03-25T19:46:11.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.071+0000 7f4ead7fa640 10 client.4429.objecter ms_dispatch 0x7f4eb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:11.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.071+0000 7f4ead7fa640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:11.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.071+0000 7f4ead7fa640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:11.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.071+0000 7f4ead7fa640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T19:46:11.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.075+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4ea80505e0 con 0x7f4eb01430c0 2024-03-25T19:46:11.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.075+0000 7f4ead7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:11.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.075+0000 7f4ead7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:11.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.323+0000 7f4eb6bc8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.323+0000 7f4eb6bc8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.323+0000 7f4eb6bc8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:46:11.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.323+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4eb0069f40 con 0x7f4eb01430c0 2024-03-25T19:46:11.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.351+0000 7f4ead7fa640 1 -- 172.21.15.130:0/544214339 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f4ea8052510 con 0x7f4eb01430c0 2024-03-25T19:46:11.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.351+0000 7f4ead7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:11.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.351+0000 7f4ead7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:11.352 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:11.352 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.353+0000 7f4eb6bc8640 10 librados: watch_flush enter 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.353+0000 7f4eb6bc8640 10 librados: watch_flush exit 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.353+0000 7f4eb6bc8640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.353+0000 7f4eb6bc8640 10 client.4429.objecter successfully canceled tick 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 >> v1:172.21.15.163:6800/1217380380 conn(0x7f4e9406dd40 legacy=0x7f4e94070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 10 monclient: shutdown 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 >> v1:172.21.15.163:6789/0 conn(0x7f4eb01430c0 legacy=0x7f4eb01d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 shutdown_connections 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 >> 172.21.15.130:0/544214339 conn(0x7f4eb00b7ea0 msgr2=0x7f4eb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:11.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 shutdown_connections 2024-03-25T19:46:11.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 -- 172.21.15.130:0/544214339 wait complete. 2024-03-25T19:46:11.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:11.354+0000 7f4eb6bc8640 1 librados: shutdown 2024-03-25T19:46:11.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:18.175 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:18.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.289+0000 7fde6e31c640 10 monclient: get_monmap_and_config 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.289+0000 7fde6e31c640 10 monclient: build_initial_monmap 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.289+0000 7fde6e31c640 10 monclient: monmap: 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:18.290197+0000 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:18.290197+0000 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:18.289 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.290+0000 7fde6e31c640 1 Processor -- start 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.290+0000 7fde6e31c640 1 -- start start 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.290+0000 7fde6e31c640 10 monclient: init 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-c con 0x7fde680a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-a con 0x7fde6813f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:18.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-b con 0x7fde681430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68142520 con 0x7fde6813f710 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680b28a0 con 0x7fde680a98f0 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680b2a30 con 0x7fde681430c0 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:18.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.291+0000 7fde6e31c640 10 monclient(hunting): authenticate will time out at 5254.436895s 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.294+0000 7fde677fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fde6813f710 0x7fde681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:58312/0 (socket says 172.21.15.130:58312) 2024-03-25T19:46:18.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.294+0000 7fde677fe640 1 -- 172.21.15.130:0/1449951725 learned_addr learned my addr 172.21.15.130:0/1449951725 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:18.302 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2084657702 0 0) 0x7fde680b28a0 con 0x7fde680a98f0 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T19:46:18.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.303+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde50003520 con 0x7fde680a98f0 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057816082 0 0) 0x7fde68142520 con 0x7fde6813f710 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 10 monclient(hunting): my global_id is 4434 2024-03-25T19:46:18.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.311+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde680b28a0 con 0x7fde6813f710 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3097869661 0 0) 0x7fde50003520 con 0x7fde680a98f0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 >> v1:172.21.15.130:6790/0 conn(0x7fde681430c0 legacy=0x7fde6814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 >> v1:172.21.15.130:6789/0 conn(0x7fde6813f710 legacy=0x7fde681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 monclient: found mon.noname-c 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680b44c0 con 0x7fde680a98f0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fde58001690 con 0x7fde680a98f0 2024-03-25T19:46:18.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde66ffd640 10 monclient: dump: 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde6e31c640 5 monclient: authenticate success, global_id 4432 2024-03-25T19:46:18.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.322+0000 7fde6e31c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:18.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde580017f0 con 0x7fde680a98f0 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 1 -- 172.21.15.130:0/1449951725 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fde58001c00 con 0x7fde680a98f0 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde66ffd640 10 monclient: dump: 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:18.353 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:18.354 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:18.354 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde6e31c640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:18.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.353+0000 7fde6e31c640 4 set_mon_vals no callback set 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 10 monclient: shutdown 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 1 -- 172.21.15.130:0/1449951725 >> v1:172.21.15.163:6789/0 conn(0x7fde680a98f0 legacy=0x7fde680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 1 -- 172.21.15.130:0/1449951725 shutdown_connections 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 1 -- 172.21.15.130:0/1449951725 >> 172.21.15.130:0/1449951725 conn(0x7fde680b7ea0 msgr2=0x7fde680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 1 -- 172.21.15.130:0/1449951725 shutdown_connections 2024-03-25T19:46:18.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.354+0000 7fde6e31c640 1 -- 172.21.15.130:0/1449951725 wait complete. 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.355+0000 7fde6e31c640 10 monclient: build_initial_monmap 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.355+0000 7fde6e31c640 10 monclient: monmap: 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.355+0000 7fde6e31c640 1 librados: starting msgr at 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.355+0000 7fde6e31c640 1 librados: starting objecter 2024-03-25T19:46:18.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.355+0000 7fde6e31c640 1 Processor -- start 2024-03-25T19:46:18.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 -- start start 2024-03-25T19:46:18.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 librados: setting wanted keys 2024-03-25T19:46:18.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 librados: calling monclient init 2024-03-25T19:46:18.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient: init 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-b con 0x7fde680a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-c con 0x7fde6813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient(hunting): picked mon.noname-a con 0x7fde681430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68138590 con 0x7fde681430c0 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde68138df0 con 0x7fde680a98f0 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde681390d0 con 0x7fde6813f710 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:18.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.356+0000 7fde6e31c640 10 monclient(hunting): authenticate will time out at 5254.502239s 2024-03-25T19:46:18.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.361+0000 7fde67fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fde680a98f0 0x7fde680b75b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:45630/0 (socket says 172.21.15.130:45630) 2024-03-25T19:46:18.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.361+0000 7fde67fff640 1 -- 172.21.15.130:0/1722791456 learned_addr learned my addr 172.21.15.130:0/1722791456 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:18.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.362+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde680a98f0 2024-03-25T19:46:18.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.362+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.364 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.365+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde6813f710 2024-03-25T19:46:18.364 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.365+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.370+0000 7fde64ff9640 10 client.?.objecter ms_handle_connect 0x7fde681430c0 2024-03-25T19:46:18.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.370+0000 7fde64ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745750850 0 0) 0x7fde681390d0 con 0x7fde6813f710 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4433 2024-03-25T19:46:18.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.371+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde6813f710 2024-03-25T19:46:18.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 542018021 0 0) 0x7fde54004eb0 con 0x7fde6813f710 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde681390d0 con 0x7fde6813f710 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fde5c001930 con 0x7fde6813f710 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 10 monclient(hunting): dump: 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:18.382 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.382+0000 7fde64ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:18.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058281898 0 0) 0x7fde68138590 con 0x7fde681430c0 2024-03-25T19:46:18.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4437 2024-03-25T19:46:18.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.384+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54004eb0 con 0x7fde681430c0 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659051807 0 0) 0x7fde68138df0 con 0x7fde680a98f0 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 10 monclient(hunting): my global_id is 4435 2024-03-25T19:46:18.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.386+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde68138590 con 0x7fde680a98f0 2024-03-25T19:46:18.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3589237658 0 0) 0x7fde681390d0 con 0x7fde6813f710 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 >> v1:172.21.15.163:6789/0 conn(0x7fde680a98f0 legacy=0x7fde680b75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 >> v1:172.21.15.130:6789/0 conn(0x7fde681430c0 legacy=0x7fde681e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 1 monclient: found mon.c 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde681393b0 con 0x7fde6813f710 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde64ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde6e31c640 5 monclient: authenticate success, global_id 4433 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.392+0000 7fde6e31c640 10 monclient: _renew_subs 2024-03-25T19:46:18.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68139690 con 0x7fde6813f710 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 10 client.4433.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 10 monclient: _renew_subs 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68137960 con 0x7fde6813f710 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 1 librados: init done 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.393+0000 7fde6e31c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.394+0000 7fde6e31c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.394+0000 7fde6e31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:18.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.394+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde680627b0 con 0x7fde6813f710 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde5c0047e0 con 0x7fde6813f710 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fde5c004af0 con 0x7fde6813f710 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde64ff9640 10 monclient: dump: 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:18.693 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.693+0000 7fde667fc640 4 set_mon_vals no callback set 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.694+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fde5c01aac0 con 0x7fde6813f710 2024-03-25T19:46:18.694 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.694+0000 7fde67fff640 10 monclient: get_auth_request con 0x7fde5406de10 auth_method 0 2024-03-25T19:46:18.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.695+0000 7fde64ff9640 10 client.4433.objecter ms_handle_connect 0x7fde5406de10 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fde5c054840 con 0x7fde6813f710 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 10 client.4433.objecter ms_dispatch 0x7fde68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 3 client.4433.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 3 client.4433.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 20 client.4433.objecter dump_active .. 0 homeless 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fde5c050180 con 0x7fde6813f710 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:18.700 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.700+0000 7fde64ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.955+0000 7fde6e31c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.955+0000 7fde6e31c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.955+0000 7fde6e31c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:18.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.955+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde6813e230 con 0x7fde6813f710 2024-03-25T19:46:18.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.976+0000 7fde64ff9640 1 -- 172.21.15.130:0/1722791456 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fde5c052280 con 0x7fde6813f710 2024-03-25T19:46:18.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.976+0000 7fde64ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:18.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.976+0000 7fde64ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:18.976 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:18.976 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 10 librados: watch_flush enter 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 10 librados: watch_flush exit 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 20 client.4433.objecter shutdown clearing up homeless session... 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 10 client.4433.objecter successfully canceled tick 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 >> v1:172.21.15.163:6800/1217380380 conn(0x7fde5406de10 legacy=0x7fde540702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 10 monclient: shutdown 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.978+0000 7fde6e31c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 >> v1:172.21.15.130:6790/0 conn(0x7fde6813f710 legacy=0x7fde68137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 shutdown_connections 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 >> 172.21.15.130:0/1722791456 conn(0x7fde680b7ea0 msgr2=0x7fde68074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:18.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 shutdown_connections 2024-03-25T19:46:18.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 -- 172.21.15.130:0/1722791456 wait complete. 2024-03-25T19:46:18.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:18.979+0000 7fde6e31c640 1 librados: shutdown 2024-03-25T19:46:18.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:20.143 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:05.076 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.148 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:05.082 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.186 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:05.119 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.201 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:05.102 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.213 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:05.114 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.215 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:05.148 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.232 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:05.165 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.235 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:05.136 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:20.254 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:05.187 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:25.795 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:25.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.909+0000 7fd1c1a1e640 10 monclient: get_monmap_and_config 2024-03-25T19:46:25.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.909+0000 7fd1c1a1e640 10 monclient: build_initial_monmap 2024-03-25T19:46:25.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.909+0000 7fd1c1a1e640 10 monclient: monmap: 2024-03-25T19:46:25.909 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:25.909 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:25.910726+0000 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:25.910726+0000 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:25.910 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.910+0000 7fd1c1a1e640 1 Processor -- start 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 1 -- start start 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient: init 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1bc061890 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1bc06e740 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1bc0ae280 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0ae830 con 0x7fd1bc06e740 2024-03-25T19:46:25.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc06f720 con 0x7fd1bc0ae280 2024-03-25T19:46:25.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc0ada80 con 0x7fd1bc061890 2024-03-25T19:46:25.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:25.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.911+0000 7fd1c1a1e640 10 monclient(hunting): authenticate will time out at 5262.057508s 2024-03-25T19:46:25.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.923+0000 7fd1bb7fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1bc0ae280 0x7fd1bc145f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:50554/0 (socket says 172.21.15.130:50554) 2024-03-25T19:46:25.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.923+0000 7fd1bb7fe640 1 -- 172.21.15.130:0/4289753849 learned_addr learned my addr 172.21.15.130:0/4289753849 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:25.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908535230 0 0) 0x7fd1bc06f720 con 0x7fd1bc0ae280 2024-03-25T19:46:25.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 10 monclient(hunting): my global_id is 4438 2024-03-25T19:46:25.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.939+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1a8003520 con 0x7fd1bc0ae280 2024-03-25T19:46:25.950 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 690944202 0 0) 0x7fd1bc0ada80 con 0x7fd1bc061890 2024-03-25T19:46:25.950 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.950 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.950 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.951 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 10 monclient(hunting): my global_id is 4436 2024-03-25T19:46:25.951 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.951+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc06f720 con 0x7fd1bc061890 2024-03-25T19:46:25.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 738873030 0 0) 0x7fd1bc0ae830 con 0x7fd1bc06e740 2024-03-25T19:46:25.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:25.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:25.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:25.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 10 monclient(hunting): my global_id is 4440 2024-03-25T19:46:25.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.954+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1bc0ada80 con 0x7fd1bc06e740 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595864664 0 0) 0x7fd1a8003520 con 0x7fd1bc0ae280 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 >> v1:172.21.15.130:6790/0 conn(0x7fd1bc061890 legacy=0x7fd1bc13f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 >> v1:172.21.15.130:6789/0 conn(0x7fd1bc06e740 legacy=0x7fd1bc145820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 monclient: found mon.noname-c 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc13fbd0 con 0x7fd1bc0ae280 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:25.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1b4001a10 con 0x7fd1bc0ae280 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1c1a1e640 5 monclient: authenticate success, global_id 4438 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1b9ffb640 10 monclient: dump: 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.961 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:25.962 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:25.962 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:25.962 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:25.962 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:25.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.960+0000 7fd1c1a1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1b4001b70 con 0x7fd1bc0ae280 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 1 -- 172.21.15.130:0/4289753849 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1b4001f80 con 0x7fd1bc0ae280 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1b9ffb640 10 monclient: dump: 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:25.975 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1c1a1e640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:25.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.975+0000 7fd1c1a1e640 4 set_mon_vals no callback set 2024-03-25T19:46:25.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 10 monclient: shutdown 2024-03-25T19:46:25.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/4289753849 >> v1:172.21.15.163:6789/0 conn(0x7fd1bc0ae280 legacy=0x7fd1bc145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/4289753849 shutdown_connections 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/4289753849 >> 172.21.15.130:0/4289753849 conn(0x7fd1bc0b0ef0 msgr2=0x7fd1bc0b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/4289753849 shutdown_connections 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.976+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/4289753849 wait complete. 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 10 monclient: build_initial_monmap 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 10 monclient: monmap: 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:25.979 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 librados: starting msgr at 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 librados: starting objecter 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 Processor -- start 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 -- start start 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 librados: setting wanted keys 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 1 librados: calling monclient init 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.977+0000 7fd1c1a1e640 10 monclient: init 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:25.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1bc061890 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1bc0ae280 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1bc1d81b0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc1d7610 con 0x7fd1bc061890 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc137c30 con 0x7fd1bc0ae280 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1bc135560 con 0x7fd1bc1d81b0 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.978+0000 7fd1c1a1e640 10 monclient(hunting): authenticate will time out at 5262.124015s 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.979+0000 7fd1bb7fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fd1bc1d81b0 0x7fd1bc135fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:39982/0 (socket says 172.21.15.130:39982) 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.979+0000 7fd1bb7fe640 1 -- 172.21.15.130:0/2027681881 learned_addr learned my addr 172.21.15.130:0/2027681881 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.979+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1bc1d81b0 2024-03-25T19:46:25.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.979+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:25.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.981+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1bc0ae280 2024-03-25T19:46:25.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:25.981+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:26.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.033+0000 7fd1b3fff640 10 client.?.objecter ms_handle_connect 0x7fd1bc061890 2024-03-25T19:46:26.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.033+0000 7fd1b3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:26.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645973258 0 0) 0x7fd1bc1d7610 con 0x7fd1bc061890 2024-03-25T19:46:26.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:26.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:26.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:26.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T19:46:26.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.053+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c004eb0 con 0x7fd1bc061890 2024-03-25T19:46:26.061 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.062+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2650148275 0 0) 0x7fd19c004eb0 con 0x7fd1bc061890 2024-03-25T19:46:26.061 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.062+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1bc1d7610 con 0x7fd1bc061890 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.066+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1a4001a70 con 0x7fd1bc061890 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.066+0000 7fd1b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.066+0000 7fd1b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.066+0000 7fd1b3fff640 10 monclient(hunting): dump: 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:26.066 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:26.067 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:26.068 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.066+0000 7fd1b3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3883202049 0 0) 0x7fd1bc1d7610 con 0x7fd1bc061890 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 >> v1:172.21.15.130:6790/0 conn(0x7fd1bc1d81b0 legacy=0x7fd1bc135fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 >> v1:172.21.15.163:6789/0 conn(0x7fd1bc0ae280 legacy=0x7fd1bc1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 1 monclient: found mon.a 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:26.075 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1bc1da9e0 con 0x7fd1bc061890 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 10 monclient: _finish_auth 0 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 5 monclient: authenticate success, global_id 4443 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 10 monclient: _renew_subs 2024-03-25T19:46:26.076 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1bc1db6a0 con 0x7fd1bc061890 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 10 monclient: _renew_subs 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1bc1368c0 con 0x7fd1bc061890 2024-03-25T19:46:26.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.075+0000 7fd1c1a1e640 1 librados: init done 2024-03-25T19:46:26.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.077+0000 7fd1c1a1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.077+0000 7fd1c1a1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.077+0000 7fd1c1a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:26.078 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.077+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1bc1dabd0 con 0x7fd1bc061890 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1a4001bd0 con 0x7fd1bc061890 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1a4003a60 con 0x7fd1bc061890 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b3fff640 10 monclient: dump: 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:26.371 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:26.372 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:26.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.371+0000 7fd1b97fa640 4 set_mon_vals no callback set 2024-03-25T19:46:26.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.372+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fd1a4019a30 con 0x7fd1bc061890 2024-03-25T19:46:26.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.372+0000 7fd1ba7fc640 10 monclient: get_auth_request con 0x7fd19c06dc60 auth_method 0 2024-03-25T19:46:26.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.372+0000 7fd1b3fff640 10 client.4443.objecter ms_handle_connect 0x7fd19c06dc60 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fd1a40536f0 con 0x7fd1bc061890 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 10 client.4443.objecter ms_dispatch 0x7fd1bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1a404f030 con 0x7fd1bc061890 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:26.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.377+0000 7fd1b3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:26.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.626+0000 7fd1c1a1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.626+0000 7fd1c1a1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.626+0000 7fd1c1a1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:26.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.626+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1bc1db830 con 0x7fd1bc061890 2024-03-25T19:46:26.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.655+0000 7fd1b3fff640 1 -- 172.21.15.130:0/2027681881 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fd1a4051130 con 0x7fd1bc061890 2024-03-25T19:46:26.655 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.655+0000 7fd1b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:26.655 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.655+0000 7fd1b3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:26.655 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:26.655 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 10 librados: watch_flush enter 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 10 librados: watch_flush exit 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 10 client.4443.objecter successfully canceled tick 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 >> v1:172.21.15.163:6800/1217380380 conn(0x7fd19c06dc60 legacy=0x7fd19c070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 10 monclient: shutdown 2024-03-25T19:46:26.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 >> v1:172.21.15.130:6789/0 conn(0x7fd1bc061890 legacy=0x7fd1bc1d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 shutdown_connections 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 >> 172.21.15.130:0/2027681881 conn(0x7fd1bc0b0ef0 msgr2=0x7fd1bc13f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 shutdown_connections 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 -- 172.21.15.130:0/2027681881 wait complete. 2024-03-25T19:46:26.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:26.658+0000 7fd1c1a1e640 1 librados: shutdown 2024-03-25T19:46:26.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:33.476 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.593+0000 7f6ac660d640 10 monclient: get_monmap_and_config 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.593+0000 7f6ac660d640 10 monclient: build_initial_monmap 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.593+0000 7f6ac660d640 10 monclient: monmap: 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:33.594256+0000 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:33.594256+0000 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:33.593 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:33.594 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:33.594 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.594+0000 7f6ac660d640 1 Processor -- start 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.594+0000 7f6ac660d640 1 -- start start 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.594+0000 7f6ac660d640 10 monclient: init 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac00a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac01430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac0142520 con 0x7f6ac013f710 2024-03-25T19:46:33.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac00b28a0 con 0x7f6ac01430c0 2024-03-25T19:46:33.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac00b2a30 con 0x7f6ac00a98f0 2024-03-25T19:46:33.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:33.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.595+0000 7f6ac660d640 10 monclient(hunting): authenticate will time out at 5269.740984s 2024-03-25T19:46:33.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.597+0000 7f6abffff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f6ac00a98f0 0x7f6ac00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:38642/0 (socket says 172.21.15.130:38642) 2024-03-25T19:46:33.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.597+0000 7f6abffff640 1 -- 172.21.15.130:0/1668882755 learned_addr learned my addr 172.21.15.130:0/1668882755 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601563909 0 0) 0x7f6ac00b2a30 con 0x7f6ac00a98f0 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 10 monclient(hunting): my global_id is 4442 2024-03-25T19:46:33.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.605+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa8003520 con 0x7f6ac00a98f0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4091094677 0 0) 0x7f6aa8003520 con 0x7f6ac00a98f0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 >> v1:172.21.15.163:6789/0 conn(0x7f6ac01430c0 legacy=0x7f6ac014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 >> v1:172.21.15.130:6789/0 conn(0x7f6ac013f710 legacy=0x7f6ac01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: _finish_hunting 0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 monclient: found mon.noname-b 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac00b44c0 con 0x7f6ac00a98f0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: _finish_auth 0 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: _check_auth_tickets 2024-03-25T19:46:33.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6ab0001690 con 0x7f6ac00a98f0 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6ac660d640 5 monclient: authenticate success, global_id 4442 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6ac660d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:46:33.624 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6abeffd640 10 monclient: dump: 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.622+0000 7f6ac660d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ab00017f0 con 0x7f6ac00a98f0 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 1 -- 172.21.15.130:0/1668882755 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6ab0001c00 con 0x7f6ac00a98f0 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6ac660d640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6ac660d640 4 set_mon_vals no callback set 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:33.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.628+0000 7f6abeffd640 10 monclient: dump: 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:33.629 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 10 monclient: shutdown 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 1 -- 172.21.15.130:0/1668882755 >> v1:172.21.15.130:6790/0 conn(0x7f6ac00a98f0 legacy=0x7f6ac00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 1 -- 172.21.15.130:0/1668882755 shutdown_connections 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 1 -- 172.21.15.130:0/1668882755 >> 172.21.15.130:0/1668882755 conn(0x7f6ac00b7ea0 msgr2=0x7f6ac00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 1 -- 172.21.15.130:0/1668882755 shutdown_connections 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.629+0000 7f6ac660d640 1 -- 172.21.15.130:0/1668882755 wait complete. 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.630+0000 7f6ac660d640 10 monclient: build_initial_monmap 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.630+0000 7f6ac660d640 10 monclient: monmap: 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:33.631 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 librados: starting msgr at 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 librados: starting objecter 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 Processor -- start 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 -- start start 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 librados: setting wanted keys 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 1 librados: calling monclient init 2024-03-25T19:46:33.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.631+0000 7f6ac660d640 10 monclient: init 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac0135590 con 0x7f6ac013f710 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.632+0000 7f6ac660d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac0136e50 con 0x7f6ac00a98f0 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.633+0000 7f6ac660d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac0135070 con 0x7f6ac01430c0 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.633+0000 7f6ac660d640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:33.634 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.633+0000 7f6ac660d640 10 monclient(hunting): authenticate will time out at 5269.778755s 2024-03-25T19:46:33.644 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.644+0000 7f6ac4baa640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f6ac01430c0 0x7f6ac01e3010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:38688/0 (socket says 172.21.15.130:38688) 2024-03-25T19:46:33.644 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.644+0000 7f6ac4baa640 1 -- 172.21.15.130:0/3190841065 learned_addr learned my addr 172.21.15.130:0/3190841065 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:33.644 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.645+0000 7f6abcff9640 10 client.?.objecter ms_handle_connect 0x7f6ac01430c0 2024-03-25T19:46:33.645 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.645+0000 7f6abcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.651+0000 7f6abcff9640 10 client.?.objecter ms_handle_connect 0x7f6ac00a98f0 2024-03-25T19:46:33.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.651+0000 7f6abcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967288046 0 0) 0x7f6ac0135070 con 0x7f6ac01430c0 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 10 monclient(hunting): my global_id is 4445 2024-03-25T19:46:33.656 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.656+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aac004eb0 con 0x7f6ac01430c0 2024-03-25T19:46:33.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.660+0000 7f6abcff9640 10 client.?.objecter ms_handle_connect 0x7f6ac013f710 2024-03-25T19:46:33.659 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.660+0000 7f6abcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3001413189 0 0) 0x7f6ac0136e50 con 0x7f6ac00a98f0 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 10 monclient(hunting): my global_id is 4447 2024-03-25T19:46:33.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.662+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ac0135070 con 0x7f6ac00a98f0 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4233435643 0 0) 0x7f6aac004eb0 con 0x7f6ac01430c0 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ac0136e50 con 0x7f6ac01430c0 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6ab8001c90 con 0x7f6ac01430c0 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 10 monclient(hunting): dump: 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.671 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.671+0000 7f6abcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.677+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1564923094 0 0) 0x7f6ac0135590 con 0x7f6ac013f710 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.677+0000 7f6abcff9640 10 monclient(hunting): _negotiate 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.677+0000 7f6abcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aac004eb0 con 0x7f6ac013f710 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3576430653 0 0) 0x7f6ac0135070 con 0x7f6ac00a98f0 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ac0135590 con 0x7f6ac00a98f0 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6ab0001b00 con 0x7f6ac00a98f0 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.678+0000 7f6abcff9640 10 monclient(hunting): dump: 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:33.678 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:33.679 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3816771636 0 0) 0x7f6ac0136e50 con 0x7f6ac01430c0 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 >> v1:172.21.15.163:6789/0 conn(0x7f6ac00a98f0 legacy=0x7f6ac0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 >> v1:172.21.15.130:6789/0 conn(0x7f6ac013f710 legacy=0x7f6ac01d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 10 monclient: _finish_hunting 0 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 1 monclient: found mon.c 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac0135d80 con 0x7f6ac01430c0 2024-03-25T19:46:33.681 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 10 monclient: _finish_auth 0 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 10 monclient: _check_auth_tickets 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6abcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 5 monclient: authenticate success, global_id 4445 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 10 monclient: _renew_subs 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac0131660 con 0x7f6ac01430c0 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 10 monclient: _renew_subs 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac00b78a0 con 0x7f6ac01430c0 2024-03-25T19:46:33.682 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.681+0000 7f6ac660d640 1 librados: init done 2024-03-25T19:46:33.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.682+0000 7f6ac660d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:33.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.682+0000 7f6ac660d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:33.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.682+0000 7f6ac660d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:33.683 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.682+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac0135f70 con 0x7f6ac01430c0 2024-03-25T19:46:33.989 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6ab8003990 con 0x7f6ac01430c0 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6ab8003bc0 con 0x7f6ac01430c0 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 10 monclient: dump: 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:33.990 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abe7fc640 4 set_mon_vals no callback set 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.990+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f6ab8003de0 con 0x7f6ac01430c0 2024-03-25T19:46:33.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.991+0000 7f6abffff640 10 monclient: get_auth_request con 0x7f6aac06dfb0 auth_method 0 2024-03-25T19:46:33.992 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.991+0000 7f6abcff9640 10 client.4445.objecter ms_handle_connect 0x7f6aac06dfb0 2024-03-25T19:46:33.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.996+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f6ab8054960 con 0x7f6ac01430c0 2024-03-25T19:46:33.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.996+0000 7f6abcff9640 10 client.4445.objecter ms_dispatch 0x7f6ac0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:33.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.996+0000 7f6abcff9640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:33.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.996+0000 7f6abcff9640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:33.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.996+0000 7f6abcff9640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-25T19:46:33.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.997+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6ab8050470 con 0x7f6ac01430c0 2024-03-25T19:46:33.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.997+0000 7f6abcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:33.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:33.997+0000 7f6abcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:34.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.258+0000 7f6ac660d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.258+0000 7f6ac660d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.258+0000 7f6ac660d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:34.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.258+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ac0135200 con 0x7f6ac01430c0 2024-03-25T19:46:34.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.301+0000 7f6abcff9640 1 -- 172.21.15.130:0/3190841065 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f6ab80523a0 con 0x7f6ac01430c0 2024-03-25T19:46:34.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.301+0000 7f6abcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:34.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.301+0000 7f6abcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:34.301 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:34.301 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 10 librados: watch_flush enter 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 10 librados: watch_flush exit 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 10 client.4445.objecter successfully canceled tick 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 >> v1:172.21.15.163:6800/1217380380 conn(0x7f6aac06dfb0 legacy=0x7f6aac070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 10 monclient: shutdown 2024-03-25T19:46:34.303 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 >> v1:172.21.15.130:6790/0 conn(0x7f6ac01430c0 legacy=0x7f6ac01e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 shutdown_connections 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 >> 172.21.15.130:0/3190841065 conn(0x7f6ac00b7ea0 msgr2=0x7f6ac0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 shutdown_connections 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 -- 172.21.15.130:0/3190841065 wait complete. 2024-03-25T19:46:34.304 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:34.303+0000 7f6ac660d640 1 librados: shutdown 2024-03-25T19:46:34.317 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:40.265 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:25.198 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.323 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:25.256 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.344 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:25.277 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.369 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:25.270 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.373 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:25.274 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.376 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:25.309 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.384 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:25.317 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.410 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:25.343 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:40.412 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:25.313 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:46:41.124 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.238+0000 7f1f2ae74640 10 monclient: get_monmap_and_config 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.238+0000 7f1f2ae74640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.238+0000 7f1f2ae74640 10 monclient: monmap: 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:41.239300+0000 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:41.239300+0000 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:41.238 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:41.239 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.239+0000 7f1f2ae74640 1 Processor -- start 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.239+0000 7f1f2ae74640 1 -- start start 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.239+0000 7f1f2ae74640 10 monclient: init 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-a con 0x7f1f240a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-b con 0x7f1f2413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-c con 0x7f1f241430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:41.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f24142520 con 0x7f1f240a98f0 2024-03-25T19:46:41.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f240b28a0 con 0x7f1f241430c0 2024-03-25T19:46:41.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f240b2a30 con 0x7f1f2413f710 2024-03-25T19:46:41.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.240+0000 7f1f2ae74640 10 monclient(hunting): authenticate will time out at 5277.386149s 2024-03-25T19:46:41.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.249+0000 7f1f29411640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1f241430c0 0x7f1f2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:49262/0 (socket says 172.21.15.130:49262) 2024-03-25T19:46:41.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.249+0000 7f1f29411640 1 -- 172.21.15.130:0/1646594986 learned_addr learned my addr 172.21.15.130:0/1646594986 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115749422 0 0) 0x7f1f240b28a0 con 0x7f1f241430c0 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 10 monclient(hunting): my global_id is 4450 2024-03-25T19:46:41.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.258+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f14003520 con 0x7f1f241430c0 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2765618262 0 0) 0x7f1f240b2a30 con 0x7f1f2413f710 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 10 monclient(hunting): my global_id is 4448 2024-03-25T19:46:41.265 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.265+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f240b28a0 con 0x7f1f2413f710 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1014120815 0 0) 0x7f1f14003520 con 0x7f1f241430c0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 >> v1:172.21.15.130:6790/0 conn(0x7f1f2413f710 legacy=0x7f1f241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 >> v1:172.21.15.130:6789/0 conn(0x7f1f240a98f0 legacy=0x7f1f240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240b44c0 con 0x7f1f241430c0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1f1c001a10 con 0x7f1f241430c0 2024-03-25T19:46:41.276 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.275+0000 7f1f237fe640 10 monclient: dump: 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.276+0000 7f1f2ae74640 5 monclient: authenticate success, global_id 4450 2024-03-25T19:46:41.277 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.276+0000 7f1f2ae74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f1c001b70 con 0x7f1f241430c0 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 1 -- 172.21.15.130:0/1646594986 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1f1c001f80 con 0x7f1f241430c0 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f2ae74640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f2ae74640 4 set_mon_vals no callback set 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.279+0000 7f1f237fe640 10 monclient: dump: 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.279 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:41.280 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 10 monclient: shutdown 2024-03-25T19:46:41.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:41.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 1 -- 172.21.15.130:0/1646594986 >> v1:172.21.15.163:6789/0 conn(0x7f1f241430c0 legacy=0x7f1f2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 1 -- 172.21.15.130:0/1646594986 shutdown_connections 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 1 -- 172.21.15.130:0/1646594986 >> 172.21.15.130:0/1646594986 conn(0x7f1f240b7ea0 msgr2=0x7f1f240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 1 -- 172.21.15.130:0/1646594986 shutdown_connections 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.280+0000 7f1f2ae74640 1 -- 172.21.15.130:0/1646594986 wait complete. 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 10 monclient: build_initial_monmap 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 10 monclient: monmap: 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:41.282 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 librados: starting msgr at 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 librados: starting objecter 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 Processor -- start 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 -- start start 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 librados: setting wanted keys 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 1 librados: calling monclient init 2024-03-25T19:46:41.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.281+0000 7f1f2ae74640 10 monclient: init 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.282+0000 7f1f2ae74640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.282+0000 7f1f2ae74640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.282+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-a con 0x7f1f240a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-c con 0x7f1f2413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 10 monclient(hunting): picked mon.noname-b con 0x7f1f241430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f241369d0 con 0x7f1f240a98f0 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f24133a60 con 0x7f1f241430c0 2024-03-25T19:46:41.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f240ab9e0 con 0x7f1f2413f710 2024-03-25T19:46:41.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:41.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.283+0000 7f1f2ae74640 10 monclient(hunting): authenticate will time out at 5277.428753s 2024-03-25T19:46:41.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.295+0000 7f1f23fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f1f2413f710 0x7f1f241362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:54882/0 (socket says 172.21.15.130:54882) 2024-03-25T19:46:41.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.295+0000 7f1f23fff640 1 -- 172.21.15.130:0/3504734065 learned_addr learned my addr 172.21.15.130:0/3504734065 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:41.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.297+0000 7f1f217fa640 10 client.?.objecter ms_handle_connect 0x7f1f2413f710 2024-03-25T19:46:41.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.297+0000 7f1f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.301+0000 7f1f217fa640 10 client.?.objecter ms_handle_connect 0x7f1f240a98f0 2024-03-25T19:46:41.301 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.301+0000 7f1f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.308+0000 7f1f217fa640 10 client.?.objecter ms_handle_connect 0x7f1f241430c0 2024-03-25T19:46:41.307 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.308+0000 7f1f217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:41.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059598000 0 0) 0x7f1f241369d0 con 0x7f1f240a98f0 2024-03-25T19:46:41.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:41.310 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:41.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:41.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 10 monclient(hunting): my global_id is 4455 2024-03-25T19:46:41.311 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.311+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f08004eb0 con 0x7f1f240a98f0 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4173056488 0 0) 0x7f1f08004eb0 con 0x7f1f240a98f0 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f241369d0 con 0x7f1f240a98f0 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1f180018c0 con 0x7f1f240a98f0 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 10 monclient(hunting): dump: 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.329 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:41.330 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:41.330 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:41.330 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:41.330 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.329+0000 7f1f217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:41.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2073721265 0 0) 0x7f1f241369d0 con 0x7f1f240a98f0 2024-03-25T19:46:41.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 >> v1:172.21.15.130:6790/0 conn(0x7f1f2413f710 legacy=0x7f1f241362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 >> v1:172.21.15.163:6789/0 conn(0x7f1f241430c0 legacy=0x7f1f24133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 1 monclient: found mon.a 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f240abcc0 con 0x7f1f240a98f0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 5 monclient: authenticate success, global_id 4455 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 10 monclient: _renew_subs 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f240abfa0 con 0x7f1f240a98f0 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 10 client.4455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:41.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 10 monclient: _renew_subs 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f24137e40 con 0x7f1f240a98f0 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.341+0000 7f1f2ae74640 1 librados: init done 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.342+0000 7f1f2ae74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.342+0000 7f1f2ae74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.342+0000 7f1f2ae74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:41.343 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.342+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f240627b0 con 0x7f1f240a98f0 2024-03-25T19:46:41.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f180035f0 con 0x7f1f240a98f0 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1f180039c0 con 0x7f1f240a98f0 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 10 monclient: dump: 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:41.642 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:41.643 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:41.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f22ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:41.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.642+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f1f18003be0 con 0x7f1f240a98f0 2024-03-25T19:46:41.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.643+0000 7f1f23fff640 10 monclient: get_auth_request con 0x7f1f0806dce0 auth_method 0 2024-03-25T19:46:41.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.643+0000 7f1f217fa640 10 client.4455.objecter ms_handle_connect 0x7f1f0806dce0 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f1f180535b0 con 0x7f1f240a98f0 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 10 client.4455.objecter ms_dispatch 0x7f1f24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 3 client.4455.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 3 client.4455.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 20 client.4455.objecter dump_active .. 0 homeless 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f1804f0c0 con 0x7f1f240a98f0 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:41.647 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.647+0000 7f1f217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:41.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.891+0000 7f1f2ae74640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:41.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.891+0000 7f1f2ae74640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:41.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.891+0000 7f1f2ae74640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:46:41.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.891+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f2413dfd0 con 0x7f1f240a98f0 2024-03-25T19:46:41.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.927+0000 7f1f217fa640 1 -- 172.21.15.130:0/3504734065 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f1f18050ff0 con 0x7f1f240a98f0 2024-03-25T19:46:41.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.927+0000 7f1f217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:41.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.927+0000 7f1f217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:41.927 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:41.927 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 10 librados: watch_flush enter 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 10 librados: watch_flush exit 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 20 client.4455.objecter shutdown clearing up homeless session... 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 10 client.4455.objecter successfully canceled tick 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 >> v1:172.21.15.163:6800/1217380380 conn(0x7f1f0806dce0 legacy=0x7f1f080701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 10 monclient: shutdown 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.929+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 >> v1:172.21.15.130:6789/0 conn(0x7f1f240a98f0 legacy=0x7f1f24132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.930+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 shutdown_connections 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.930+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 >> 172.21.15.130:0/3504734065 conn(0x7f1f240b7ea0 msgr2=0x7f1f24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.930+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 shutdown_connections 2024-03-25T19:46:41.931 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.930+0000 7f1f2ae74640 1 -- 172.21.15.130:0/3504734065 wait complete. 2024-03-25T19:46:41.931 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:41.930+0000 7f1f2ae74640 1 librados: shutdown 2024-03-25T19:46:41.941 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:48.747 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.863+0000 7f945eaa8640 10 monclient: get_monmap_and_config 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.863+0000 7f945eaa8640 10 monclient: build_initial_monmap 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.863+0000 7f945eaa8640 10 monclient: monmap: 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:48.864382+0000 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:48.864382+0000 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:48.863 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:48.864 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:48.864 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:48.864 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:48.864 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.864+0000 7f945eaa8640 1 Processor -- start 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.864+0000 7f945eaa8640 1 -- start start 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.864+0000 7f945eaa8640 10 monclient: init 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f94580a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f945813f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f94581430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9458142520 con 0x7f94581430c0 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580b28a0 con 0x7f94580a98f0 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580b2a30 con 0x7f945813f710 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:48.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.865+0000 7f945eaa8640 10 monclient(hunting): authenticate will time out at 5285.011146s 2024-03-25T19:46:48.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.879+0000 7f945d045640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f94581430c0 0x7f945814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56586/0 (socket says 172.21.15.130:56586) 2024-03-25T19:46:48.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.879+0000 7f945d045640 1 -- 172.21.15.130:0/2705076056 learned_addr learned my addr 172.21.15.130:0/2705076056 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025825745 0 0) 0x7f94580b28a0 con 0x7f94580a98f0 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 10 monclient(hunting): my global_id is 4453 2024-03-25T19:46:48.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.890+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9448003520 con 0x7f94580a98f0 2024-03-25T19:46:48.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3386345960 0 0) 0x7f9458142520 con 0x7f94581430c0 2024-03-25T19:46:48.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:48.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:48.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:48.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T19:46:48.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.894+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94580b28a0 con 0x7f94581430c0 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958573541 0 0) 0x7f94580b2a30 con 0x7f945813f710 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 10 monclient(hunting): _negotiate 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 10 monclient(hunting): my global_id is 4451 2024-03-25T19:46:48.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.896+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9458142520 con 0x7f945813f710 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 414481764 0 0) 0x7f9448003520 con 0x7f94580a98f0 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 >> v1:172.21.15.130:6790/0 conn(0x7f945813f710 legacy=0x7f94581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 >> v1:172.21.15.130:6789/0 conn(0x7f94581430c0 legacy=0x7f945814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: _finish_hunting 0 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 monclient: found mon.noname-c 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:48.900 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94580b44c0 con 0x7f94580a98f0 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: _finish_auth 0 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: _check_auth_tickets 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f945eaa8640 5 monclient: authenticate success, global_id 4453 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f945eaa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f944c001650 con 0x7f94580a98f0 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f94577fe640 10 monclient: dump: 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.901 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:48.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.900+0000 7f945eaa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:48.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f944c0018a0 con 0x7f94580a98f0 2024-03-25T19:46:48.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:48.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 1 -- 172.21.15.130:0/2705076056 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f944c001c30 con 0x7f94580a98f0 2024-03-25T19:46:48.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f945eaa8640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:48.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f945eaa8640 4 set_mon_vals no callback set 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.923+0000 7f94577fe640 10 monclient: dump: 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:48.924 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.924+0000 7f945eaa8640 10 monclient: shutdown 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.924+0000 7f945eaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.924+0000 7f945eaa8640 1 -- 172.21.15.130:0/2705076056 >> v1:172.21.15.163:6789/0 conn(0x7f94580a98f0 legacy=0x7f94580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.924+0000 7f945eaa8640 1 -- 172.21.15.130:0/2705076056 shutdown_connections 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.925+0000 7f945eaa8640 1 -- 172.21.15.130:0/2705076056 >> 172.21.15.130:0/2705076056 conn(0x7f94580b7ea0 msgr2=0x7f94580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.925+0000 7f945eaa8640 1 -- 172.21.15.130:0/2705076056 shutdown_connections 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.925+0000 7f945eaa8640 1 -- 172.21.15.130:0/2705076056 wait complete. 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 10 monclient: build_initial_monmap 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 10 monclient: monmap: 2024-03-25T19:46:48.926 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 librados: starting msgr at 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 librados: starting objecter 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 Processor -- start 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 -- start start 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 librados: setting wanted keys 2024-03-25T19:46:48.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 1 librados: calling monclient init 2024-03-25T19:46:48.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.926+0000 7f945eaa8640 10 monclient: init 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.927+0000 7f945eaa8640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.927+0000 7f945eaa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.927+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-c con 0x7f94580a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.927+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-b con 0x7f945813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 10 monclient(hunting): picked mon.noname-a con 0x7f94581430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94580abea0 con 0x7f94581430c0 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9458138f00 con 0x7f945813f710 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9458139970 con 0x7f94580a98f0 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:48.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.928+0000 7f945eaa8640 10 monclient(hunting): authenticate will time out at 5285.073719s 2024-03-25T19:46:48.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.929+0000 7f945c844640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f94580a98f0 0x7f94581386c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:57268/0 (socket says 172.21.15.130:57268) 2024-03-25T19:46:48.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.929+0000 7f945c844640 1 -- 172.21.15.130:0/740321595 learned_addr learned my addr 172.21.15.130:0/740321595 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:48.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.932+0000 7f94557fa640 10 client.?.objecter ms_handle_connect 0x7f94580a98f0 2024-03-25T19:46:48.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.932+0000 7f94557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1624071386 0 0) 0x7f9458139970 con 0x7f94580a98f0 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 monclient(hunting): my global_id is 4454 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f943c004eb0 con 0x7f94580a98f0 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 client.?.objecter ms_handle_connect 0x7f94581430c0 2024-03-25T19:46:48.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.937+0000 7f94557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:48.947 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.948+0000 7f94557fa640 10 client.?.objecter ms_handle_connect 0x7f945813f710 2024-03-25T19:46:48.947 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.948+0000 7f94557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185293683 0 0) 0x7f94580abea0 con 0x7f94581430c0 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 10 monclient(hunting): _negotiate 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 10 monclient(hunting): my global_id is 4461 2024-03-25T19:46:48.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.949+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9458139970 con 0x7f94581430c0 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2668579865 0 0) 0x7f943c004eb0 con 0x7f94580a98f0 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94580abea0 con 0x7f94580a98f0 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f944c001c70 con 0x7f94580a98f0 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 10 monclient(hunting): dump: 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:48.953 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:48.954 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:48.954 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:48.954 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:48.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.953+0000 7f94557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3597182495 0 0) 0x7f94580abea0 con 0x7f94580a98f0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 >> v1:172.21.15.163:6789/0 conn(0x7f945813f710 legacy=0x7f9458136400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 >> v1:172.21.15.130:6789/0 conn(0x7f94581430c0 legacy=0x7f94580ac2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 10 monclient: _finish_hunting 0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 1 monclient: found mon.c 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9458137b40 con 0x7f94580a98f0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 10 monclient: _finish_auth 0 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 10 monclient: _check_auth_tickets 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f94557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 5 monclient: authenticate success, global_id 4454 2024-03-25T19:46:48.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 10 monclient: _renew_subs 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94581e34b0 con 0x7f94580a98f0 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 10 client.4454.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 10 monclient: _renew_subs 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94581e3820 con 0x7f94580a98f0 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.961+0000 7f945eaa8640 1 librados: init done 2024-03-25T19:46:48.962 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.962+0000 7f945eaa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:48.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.962+0000 7f945eaa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:48.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.962+0000 7f945eaa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:48.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:48.962+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9458137d30 con 0x7f94580a98f0 2024-03-25T19:46:49.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f944c001230 con 0x7f94580a98f0 2024-03-25T19:46:49.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:49.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f944c003b00 con 0x7f94580a98f0 2024-03-25T19:46:49.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f94557fa640 10 monclient: dump: 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:49.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.253+0000 7f9456ffd640 4 set_mon_vals no callback set 2024-03-25T19:46:49.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.254+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f944c003d20 con 0x7f94580a98f0 2024-03-25T19:46:49.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.255+0000 7f9457fff640 10 monclient: get_auth_request con 0x7f943c06dd10 auth_method 0 2024-03-25T19:46:49.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.255+0000 7f94557fa640 10 client.4454.objecter ms_handle_connect 0x7f943c06dd10 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f944c0548f0 con 0x7f94580a98f0 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 10 client.4454.objecter ms_dispatch 0x7f9458000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 3 client.4454.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 3 client.4454.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 20 client.4454.objecter dump_active .. 0 homeless 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f944c050400 con 0x7f94580a98f0 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:49.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.261+0000 7f94557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:49.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.495+0000 7f945eaa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.495+0000 7f945eaa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.495+0000 7f945eaa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:49.495 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.495+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94580abca0 con 0x7f94580a98f0 2024-03-25T19:46:49.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.518+0000 7f94557fa640 1 -- 172.21.15.130:0/740321595 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f944c052330 con 0x7f94580a98f0 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.518+0000 7f94557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.518+0000 7f94557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:49.518 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:49.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 10 librados: watch_flush enter 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 10 librados: watch_flush exit 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 20 client.4454.objecter shutdown clearing up homeless session... 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 10 client.4454.objecter successfully canceled tick 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 >> v1:172.21.15.163:6800/1217380380 conn(0x7f943c06dd10 legacy=0x7f943c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 10 monclient: shutdown 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 >> v1:172.21.15.130:6790/0 conn(0x7f94580a98f0 legacy=0x7f94581386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 shutdown_connections 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 >> 172.21.15.130:0/740321595 conn(0x7f94580b7ea0 msgr2=0x7f9458074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 shutdown_connections 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 -- 172.21.15.130:0/740321595 wait complete. 2024-03-25T19:46:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:49.520+0000 7f945eaa8640 1 librados: shutdown 2024-03-25T19:46:49.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:46:56.337 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.455+0000 7f6e4f42a640 10 monclient: get_monmap_and_config 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.455+0000 7f6e4f42a640 10 monclient: build_initial_monmap 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.455+0000 7f6e4f42a640 10 monclient: monmap: 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:46:56.456453+0000 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:46:56.456453+0000 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:56.455 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.456 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:56.456 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:46:56.456 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:46:56.456 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.457 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.456+0000 7f6e4f42a640 1 Processor -- start 2024-03-25T19:46:56.457 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.456+0000 7f6e4f42a640 1 -- start start 2024-03-25T19:46:56.457 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.456+0000 7f6e4f42a640 10 monclient: init 2024-03-25T19:46:56.457 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:56.457 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-a con 0x7f6e480a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-c con 0x7f6e4813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-b con 0x7f6e481430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e48142520 con 0x7f6e480a98f0 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e480b28a0 con 0x7f6e4813f710 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e480b2a30 con 0x7f6e481430c0 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:56.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.457+0000 7f6e4f42a640 10 monclient(hunting): authenticate will time out at 5292.603260s 2024-03-25T19:46:56.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.469+0000 7f6e4c9c5640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6e4813f710 0x7f6e481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:60642/0 (socket says 172.21.15.130:60642) 2024-03-25T19:46:56.468 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.469+0000 7f6e4c9c5640 1 -- 172.21.15.130:0/1278330199 learned_addr learned my addr 172.21.15.130:0/1278330199 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:56.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1349559814 0 0) 0x7f6e480b28a0 con 0x7f6e4813f710 2024-03-25T19:46:56.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 10 monclient(hunting): my global_id is 4459 2024-03-25T19:46:56.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.486+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e34003520 con 0x7f6e4813f710 2024-03-25T19:46:56.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441739609 0 0) 0x7f6e480b2a30 con 0x7f6e481430c0 2024-03-25T19:46:56.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.491 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 10 monclient(hunting): my global_id is 4457 2024-03-25T19:46:56.492 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.492+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e480b28a0 con 0x7f6e481430c0 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1489937778 0 0) 0x7f6e48142520 con 0x7f6e480a98f0 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 10 monclient(hunting): my global_id is 4464 2024-03-25T19:46:56.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.498+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e480b2a30 con 0x7f6e480a98f0 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 789771433 0 0) 0x7f6e34003520 con 0x7f6e4813f710 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 >> v1:172.21.15.130:6790/0 conn(0x7f6e481430c0 legacy=0x7f6e4814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 >> v1:172.21.15.130:6789/0 conn(0x7f6e480a98f0 legacy=0x7f6e480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 monclient: found mon.noname-c 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:46:56.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e480b44c0 con 0x7f6e4813f710 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: _finish_auth 0 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e38001690 con 0x7f6e4813f710 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e3ffff640 10 monclient: dump: 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.508 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.509 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e4f42a640 5 monclient: authenticate success, global_id 4459 2024-03-25T19:46:56.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.507+0000 7f6e4f42a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e380017f0 con 0x7f6e4813f710 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 1 -- 172.21.15.130:0/1278330199 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e38001c00 con 0x7f6e4813f710 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e4f42a640 10 monclient: get_monmap_and_config success 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e4f42a640 4 set_mon_vals no callback set 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.513+0000 7f6e3ffff640 10 monclient: dump: 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.513 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.514 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 10 monclient: shutdown 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 1 -- 172.21.15.130:0/1278330199 >> v1:172.21.15.163:6789/0 conn(0x7f6e4813f710 legacy=0x7f6e481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 1 -- 172.21.15.130:0/1278330199 shutdown_connections 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 1 -- 172.21.15.130:0/1278330199 >> 172.21.15.130:0/1278330199 conn(0x7f6e480b7ea0 msgr2=0x7f6e480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 1 -- 172.21.15.130:0/1278330199 shutdown_connections 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.514+0000 7f6e4f42a640 1 -- 172.21.15.130:0/1278330199 wait complete. 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 10 monclient: build_initial_monmap 2024-03-25T19:46:56.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 10 monclient: monmap: 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 librados: starting msgr at 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 librados: starting objecter 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 Processor -- start 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 -- start start 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 librados: setting wanted keys 2024-03-25T19:46:56.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 1 librados: calling monclient init 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.515+0000 7f6e4f42a640 10 monclient: init 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient: _reopen_session rank -1 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-a con 0x7f6e480a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-b con 0x7f6e4813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient(hunting): picked mon.noname-c con 0x7f6e481430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e48138a70 con 0x7f6e480a98f0 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e480aba50 con 0x7f6e4813f710 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e481d77a0 con 0x7f6e481430c0 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient(hunting): _renew_subs 2024-03-25T19:46:56.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.516+0000 7f6e4f42a640 10 monclient(hunting): authenticate will time out at 5292.662110s 2024-03-25T19:46:56.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.527+0000 7f6e4c9c5640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6e4813f710 0x7f6e481d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:60672/0 (socket says 172.21.15.130:60672) 2024-03-25T19:46:56.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.527+0000 7f6e4c9c5640 1 -- 172.21.15.130:0/185946419 learned_addr learned my addr 172.21.15.130:0/185946419 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:46:56.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.529+0000 7f6e3dffb640 10 client.?.objecter ms_handle_connect 0x7f6e4813f710 2024-03-25T19:46:56.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.529+0000 7f6e3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.532+0000 7f6e3dffb640 10 client.?.objecter ms_handle_connect 0x7f6e481430c0 2024-03-25T19:46:56.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.532+0000 7f6e3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457004306 0 0) 0x7f6e480aba50 con 0x7f6e4813f710 2024-03-25T19:46:56.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 10 monclient(hunting): my global_id is 4462 2024-03-25T19:46:56.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.544+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e2c004eb0 con 0x7f6e4813f710 2024-03-25T19:46:56.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.546+0000 7f6e3dffb640 10 client.?.objecter ms_handle_connect 0x7f6e480a98f0 2024-03-25T19:46:56.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.546+0000 7f6e3dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:46:56.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1432843893 0 0) 0x7f6e481d77a0 con 0x7f6e481430c0 2024-03-25T19:46:56.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 10 monclient(hunting): my global_id is 4460 2024-03-25T19:46:56.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.547+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e480aba50 con 0x7f6e481430c0 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.555+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277515517 0 0) 0x7f6e48138a70 con 0x7f6e480a98f0 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.555+0000 7f6e3dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.555+0000 7f6e3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.555+0000 7f6e3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.555+0000 7f6e3dffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T19:46:56.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.556+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e481d77a0 con 0x7f6e480a98f0 2024-03-25T19:46:56.565 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931733972 0 0) 0x7f6e480aba50 con 0x7f6e481430c0 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e48138a70 con 0x7f6e481430c0 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e40001b50 con 0x7f6e481430c0 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 10 monclient(hunting): dump: 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.566 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.566+0000 7f6e3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:46:56.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.569+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3663067948 0 0) 0x7f6e2c004eb0 con 0x7f6e4813f710 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.569+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e480aba50 con 0x7f6e4813f710 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.569+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e38001bc0 con 0x7f6e4813f710 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.569+0000 7f6e3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.569+0000 7f6e3dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 10 monclient(hunting): dump: 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.570 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 706098776 0 0) 0x7f6e481d77a0 con 0x7f6e480a98f0 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6e2c004eb0 con 0x7f6e480a98f0 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e44001830 con 0x7f6e480a98f0 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.570+0000 7f6e3dffb640 10 monclient(hunting): dump: 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.571 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4009871243 0 0) 0x7f6e48138a70 con 0x7f6e481430c0 2024-03-25T19:46:56.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 >> v1:172.21.15.163:6789/0 conn(0x7f6e4813f710 legacy=0x7f6e481d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 >> v1:172.21.15.130:6789/0 conn(0x7f6e480a98f0 legacy=0x7f6e48138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:56.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:46:56.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 1 monclient: found mon.c 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e481d83d0 con 0x7f6e481430c0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 10 monclient: _finish_auth 0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e4f42a640 5 monclient: authenticate success, global_id 4460 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e4f42a640 10 monclient: _renew_subs 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e4f42a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.577+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e481d41e0 con 0x7f6e481430c0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.578+0000 7f6e4f42a640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.578+0000 7f6e4f42a640 10 monclient: _renew_subs 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.578+0000 7f6e4f42a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.578+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e481d5110 con 0x7f6e481430c0 2024-03-25T19:46:56.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.578+0000 7f6e4f42a640 1 librados: init done 2024-03-25T19:46:56.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.579+0000 7f6e4f42a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:56.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.579+0000 7f6e4f42a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:56.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.579+0000 7f6e4f42a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:56.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.579+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e4814f320 con 0x7f6e481430c0 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6e40002060 con 0x7f6e481430c0 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6e40003b10 con 0x7f6e481430c0 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3dffb640 10 monclient: dump: 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:46:56.889 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:46:56.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.889+0000 7f6e3f7fe640 4 set_mon_vals no callback set 2024-03-25T19:46:56.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.890+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f6e40019b20 con 0x7f6e481430c0 2024-03-25T19:46:56.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.890+0000 7f6e4d1c6640 10 monclient: get_auth_request con 0x7f6e2c06e010 auth_method 0 2024-03-25T19:46:56.891 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.891+0000 7f6e3dffb640 10 client.4460.objecter ms_handle_connect 0x7f6e2c06e010 2024-03-25T19:46:56.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f6e400538a0 con 0x7f6e481430c0 2024-03-25T19:46:56.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 10 client.4460.objecter ms_dispatch 0x7f6e48000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:46:56.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:46:56.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:46:56.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T19:46:56.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6e4004f1e0 con 0x7f6e481430c0 2024-03-25T19:46:56.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:46:56.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:56.896+0000 7f6e3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:46:57.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.138+0000 7f6e4f42a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.138+0000 7f6e4f42a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.138+0000 7f6e4f42a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:46:57.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.138+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e481d7930 con 0x7f6e481430c0 2024-03-25T19:46:57.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.210+0000 7f6e3dffb640 1 -- 172.21.15.130:0/185946419 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f6e400512e0 con 0x7f6e481430c0 2024-03-25T19:46:57.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.210+0000 7f6e3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:46:57.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.210+0000 7f6e3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:46:57.210 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:46:57.211 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 10 librados: watch_flush enter 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 10 librados: watch_flush exit 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 10 client.4460.objecter successfully canceled tick 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 >> v1:172.21.15.163:6800/1217380380 conn(0x7f6e2c06e010 legacy=0x7f6e2c0704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 10 monclient: shutdown 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:46:57.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 >> v1:172.21.15.130:6790/0 conn(0x7f6e481430c0 legacy=0x7f6e481d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:46:57.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 shutdown_connections 2024-03-25T19:46:57.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 >> 172.21.15.130:0/185946419 conn(0x7f6e480b7ea0 msgr2=0x7f6e4813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:46:57.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 shutdown_connections 2024-03-25T19:46:57.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 -- 172.21.15.130:0/185946419 wait complete. 2024-03-25T19:46:57.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:46:57.213+0000 7f6e4f42a640 1 librados: shutdown 2024-03-25T19:46:57.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:04.031 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:04.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.133+0000 7f8a89302640 10 monclient: get_monmap_and_config 2024-03-25T19:47:04.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.134+0000 7f8a89302640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.134+0000 7f8a89302640 10 monclient: monmap: 2024-03-25T19:47:04.134 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:04.135867+0000 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:04.135867+0000 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:04.135 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.134+0000 7f8a89302640 1 Processor -- start 2024-03-25T19:47:04.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.134+0000 7f8a89302640 1 -- start start 2024-03-25T19:47:04.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.134+0000 7f8a89302640 10 monclient: init 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-c con 0x7f8a840a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-a con 0x7f8a8413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-b con 0x7f8a841430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a84142520 con 0x7f8a8413f710 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a840b28a0 con 0x7f8a840a98f0 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a840b2a30 con 0x7f8a841430c0 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.135+0000 7f8a89302640 10 monclient(hunting): authenticate will time out at 5300.282276s 2024-03-25T19:47:04.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.144+0000 7f8a82ffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8a840a98f0 0x7f8a840a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:44228/0 (socket says 172.21.15.130:44228) 2024-03-25T19:47:04.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.144+0000 7f8a82ffd640 1 -- 172.21.15.130:0/594087191 learned_addr learned my addr 172.21.15.130:0/594087191 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:04.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668914879 0 0) 0x7f8a840b28a0 con 0x7f8a840a98f0 2024-03-25T19:47:04.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 10 monclient(hunting): my global_id is 4465 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.159+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a70003520 con 0x7f8a840a98f0 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813294008 0 0) 0x7f8a840b2a30 con 0x7f8a841430c0 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 10 monclient(hunting): my global_id is 4463 2024-03-25T19:47:04.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.160+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a840b28a0 con 0x7f8a841430c0 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2486493664 0 0) 0x7f8a840b28a0 con 0x7f8a841430c0 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 >> v1:172.21.15.163:6789/0 conn(0x7f8a840a98f0 legacy=0x7f8a840a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 >> v1:172.21.15.130:6789/0 conn(0x7f8a8413f710 legacy=0x7f8a841429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 monclient: found mon.noname-b 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840b44c0 con 0x7f8a841430c0 2024-03-25T19:47:04.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8a7c0019f0 con 0x7f8a841430c0 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a89302640 5 monclient: authenticate success, global_id 4463 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a89302640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a827fc640 10 monclient: dump: 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:04.169 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:04.170 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:04.170 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:04.170 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.170 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.167+0000 7f8a89302640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:04.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a7c001b50 con 0x7f8a841430c0 2024-03-25T19:47:04.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 1 -- 172.21.15.130:0/594087191 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8a7c001f20 con 0x7f8a841430c0 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.194+0000 7f8a827fc640 10 monclient: dump: 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:04.196 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:04.197 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:04.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 4 set_mon_vals no callback set 2024-03-25T19:47:04.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 10 monclient: shutdown 2024-03-25T19:47:04.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:04.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 1 -- 172.21.15.130:0/594087191 >> v1:172.21.15.130:6790/0 conn(0x7f8a841430c0 legacy=0x7f8a8414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 1 -- 172.21.15.130:0/594087191 shutdown_connections 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.195+0000 7f8a89302640 1 -- 172.21.15.130:0/594087191 >> 172.21.15.130:0/594087191 conn(0x7f8a840b7ea0 msgr2=0x7f8a840b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.196+0000 7f8a89302640 1 -- 172.21.15.130:0/594087191 shutdown_connections 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.196+0000 7f8a89302640 1 -- 172.21.15.130:0/594087191 wait complete. 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.196+0000 7f8a89302640 10 monclient: build_initial_monmap 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.196+0000 7f8a89302640 10 monclient: monmap: 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:04.199 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 librados: starting msgr at 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 librados: starting objecter 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 Processor -- start 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 -- start start 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 librados: setting wanted keys 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 librados: calling monclient init 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient: init 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-b con 0x7f8a840a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-c con 0x7f8a8413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient(hunting): picked mon.noname-a con 0x7f8a841430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:04.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a840b02b0 con 0x7f8a841430c0 2024-03-25T19:47:04.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a840b0500 con 0x7f8a840a98f0 2024-03-25T19:47:04.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a840aea50 con 0x7f8a8413f710 2024-03-25T19:47:04.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:04.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.197+0000 7f8a89302640 10 monclient(hunting): authenticate will time out at 5300.344421s 2024-03-25T19:47:04.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.205+0000 7f8a82ffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8a840a98f0 0x7f8a840ae180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:44272/0 (socket says 172.21.15.130:44272) 2024-03-25T19:47:04.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.205+0000 7f8a82ffd640 1 -- 172.21.15.130:0/2219247005 learned_addr learned my addr 172.21.15.130:0/2219247005 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:04.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.205+0000 7f8a7b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a840a98f0 2024-03-25T19:47:04.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.205+0000 7f8a7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.217+0000 7f8a7b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a841430c0 2024-03-25T19:47:04.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.217+0000 7f8a7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.217+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235969965 0 0) 0x7f8a840b0500 con 0x7f8a840a98f0 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.218+0000 7f8a7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.218+0000 7f8a7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.218+0000 7f8a7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.218+0000 7f8a7b7fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T19:47:04.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.218+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a64004eb0 con 0x7f8a840a98f0 2024-03-25T19:47:04.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.228+0000 7f8a7b7fe640 10 client.?.objecter ms_handle_connect 0x7f8a8413f710 2024-03-25T19:47:04.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.228+0000 7f8a7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:04.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925990602 0 0) 0x7f8a64004eb0 con 0x7f8a840a98f0 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a840b0500 con 0x7f8a840a98f0 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8a74001990 con 0x7f8a840a98f0 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 10 monclient(hunting): dump: 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:04.231 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:04.232 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.230+0000 7f8a7b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1951690667 0 0) 0x7f8a840b0500 con 0x7f8a840a98f0 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 >> v1:172.21.15.130:6790/0 conn(0x7f8a8413f710 legacy=0x7f8a840aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 >> v1:172.21.15.130:6789/0 conn(0x7f8a841430c0 legacy=0x7f8a84133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 1 monclient: found mon.b 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:47:04.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840af680 con 0x7f8a840a98f0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 5 monclient: authenticate success, global_id 4468 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 10 monclient: _renew_subs 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a84133780 con 0x7f8a840a98f0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 10 monclient: _renew_subs 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a840aba90 con 0x7f8a840a98f0 2024-03-25T19:47:04.238 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.236+0000 7f8a89302640 1 librados: init done 2024-03-25T19:47:04.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.237+0000 7f8a89302640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.237+0000 7f8a89302640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.237+0000 7f8a89302640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:47:04.239 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.237+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a84133b20 con 0x7f8a840a98f0 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a740047e0 con 0x7f8a840a98f0 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8a74004a10 con 0x7f8a840a98f0 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a7b7fe640 10 monclient: dump: 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:04.516 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:04.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.515+0000 7f8a81ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:04.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.516+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8a74004c30 con 0x7f8a840a98f0 2024-03-25T19:47:04.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.516+0000 7f8a7bfff640 10 monclient: get_auth_request con 0x7f8a6406dc70 auth_method 0 2024-03-25T19:47:04.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.517+0000 7f8a7b7fe640 10 client.4468.objecter ms_handle_connect 0x7f8a6406dc70 2024-03-25T19:47:04.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8a74054700 con 0x7f8a840a98f0 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 10 client.4468.objecter ms_dispatch 0x7f8a84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a74050210 con 0x7f8a840a98f0 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:04.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.521+0000 7f8a7b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:04.774 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.774+0000 7f8a89302640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:04.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.774+0000 7f8a89302640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:04.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.774+0000 7f8a89302640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:47:04.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.774+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a840aebe0 con 0x7f8a840a98f0 2024-03-25T19:47:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.812+0000 7f8a7b7fe640 1 -- 172.21.15.130:0/2219247005 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8a74052140 con 0x7f8a840a98f0 2024-03-25T19:47:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.812+0000 7f8a7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.812+0000 7f8a7b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:04.813 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:04.813 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:04.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.815+0000 7f8a89302640 10 librados: watch_flush enter 2024-03-25T19:47:04.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.815+0000 7f8a89302640 10 librados: watch_flush exit 2024-03-25T19:47:04.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.815+0000 7f8a89302640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T19:47:04.816 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.815+0000 7f8a89302640 10 client.4468.objecter successfully canceled tick 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 >> v1:172.21.15.163:6800/1217380380 conn(0x7f8a6406dc70 legacy=0x7f8a64070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 10 monclient: shutdown 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 >> v1:172.21.15.163:6789/0 conn(0x7f8a840a98f0 legacy=0x7f8a840ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 shutdown_connections 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 >> 172.21.15.130:0/2219247005 conn(0x7f8a840b7ea0 msgr2=0x7f8a84074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 shutdown_connections 2024-03-25T19:47:04.817 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 -- 172.21.15.130:0/2219247005 wait complete. 2024-03-25T19:47:04.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:04.816+0000 7f8a89302640 1 librados: shutdown 2024-03-25T19:47:04.829 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:05.261 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:50.194 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.310 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:50.243 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.313 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:50.246 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.313 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:50.214 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.317 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:50.218 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.317 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:50.250 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.336 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:05:50.269 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.348 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:05:50.282 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:05.356 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:05:50.257 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:11.636 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:11.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.752+0000 7f09638e6640 10 monclient: get_monmap_and_config 2024-03-25T19:47:11.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.752+0000 7f09638e6640 10 monclient: build_initial_monmap 2024-03-25T19:47:11.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.752+0000 7f09638e6640 10 monclient: monmap: 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:11.753956+0000 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:11.753956+0000 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:11.753 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.752+0000 7f09638e6640 1 Processor -- start 2024-03-25T19:47:11.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 1 -- start start 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient: init 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-b con 0x7f095c0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-a con 0x7f095c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-c con 0x7f095c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.753+0000 7f09638e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c142520 con 0x7f095c13f710 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.754+0000 7f09638e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0b28a0 con 0x7f095c1430c0 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.754+0000 7f09638e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c0b2a30 con 0x7f095c0a98f0 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.754+0000 7f09638e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:11.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.754+0000 7f09638e6640 10 monclient(hunting): authenticate will time out at 5307.900651s 2024-03-25T19:47:11.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.761+0000 7f0960e81640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f095c13f710 0x7f095c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:53320/0 (socket says 172.21.15.130:53320) 2024-03-25T19:47:11.762 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.761+0000 7f0960e81640 1 -- 172.21.15.130:0/1985754665 learned_addr learned my addr 172.21.15.130:0/1985754665 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:11.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2916740384 0 0) 0x7f095c142520 con 0x7f095c13f710 2024-03-25T19:47:11.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:11.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:11.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:11.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 10 monclient(hunting): my global_id is 4476 2024-03-25T19:47:11.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.767+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0948003520 con 0x7f095c13f710 2024-03-25T19:47:11.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.775+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748207450 0 0) 0x7f095c0b2a30 con 0x7f095c0a98f0 2024-03-25T19:47:11.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.776+0000 7f0953fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:11.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.776+0000 7f0953fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:11.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.776+0000 7f0953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:11.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.776+0000 7f0953fff640 10 monclient(hunting): my global_id is 4469 2024-03-25T19:47:11.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.776+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c142520 con 0x7f095c0a98f0 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1881411609 0 0) 0x7f0948003520 con 0x7f095c13f710 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 >> v1:172.21.15.130:6790/0 conn(0x7f095c0a98f0 legacy=0x7f095c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 >> v1:172.21.15.163:6789/0 conn(0x7f095c1430c0 legacy=0x7f095c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 monclient: found mon.noname-a 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c0b44c0 con 0x7f095c13f710 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:11.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f094c001690 con 0x7f095c13f710 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f09638e6640 5 monclient: authenticate success, global_id 4476 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f09638e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f0953fff640 10 monclient: dump: 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:11.780 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.778+0000 7f09638e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f094c0017f0 con 0x7f095c13f710 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 1 -- 172.21.15.130:0/1985754665 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f094c001c00 con 0x7f095c13f710 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f0953fff640 10 monclient: dump: 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:11.786 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f09638e6640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:11.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.785+0000 7f09638e6640 4 set_mon_vals no callback set 2024-03-25T19:47:11.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 10 monclient: shutdown 2024-03-25T19:47:11.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:11.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 1 -- 172.21.15.130:0/1985754665 >> v1:172.21.15.130:6789/0 conn(0x7f095c13f710 legacy=0x7f095c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:11.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 1 -- 172.21.15.130:0/1985754665 shutdown_connections 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 1 -- 172.21.15.130:0/1985754665 >> 172.21.15.130:0/1985754665 conn(0x7f095c0b7ea0 msgr2=0x7f095c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 1 -- 172.21.15.130:0/1985754665 shutdown_connections 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.786+0000 7f09638e6640 1 -- 172.21.15.130:0/1985754665 wait complete. 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.787+0000 7f09638e6640 10 monclient: build_initial_monmap 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.787+0000 7f09638e6640 10 monclient: monmap: 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:11.790 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.787+0000 7f09638e6640 1 librados: starting msgr at 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.787+0000 7f09638e6640 1 librados: starting objecter 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 Processor -- start 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 -- start start 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 librados: setting wanted keys 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 librados: calling monclient init 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient: init 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-a con 0x7f095c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-c con 0x7f095c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:11.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient(hunting): picked mon.noname-b con 0x7f095c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:11.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c1397c0 con 0x7f095c0a98f0 2024-03-25T19:47:11.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c1daa20 con 0x7f095c1430c0 2024-03-25T19:47:11.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095c1d8210 con 0x7f095c13f710 2024-03-25T19:47:11.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:11.792 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.788+0000 7f09638e6640 10 monclient(hunting): authenticate will time out at 5307.935411s 2024-03-25T19:47:11.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.798+0000 7f0960e81640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f095c13f710 0x7f095c1d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:49980/0 (socket says 172.21.15.130:49980) 2024-03-25T19:47:11.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.798+0000 7f0960e81640 1 -- 172.21.15.130:0/4158397285 learned_addr learned my addr 172.21.15.130:0/4158397285 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:11.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.798+0000 7f0951ffb640 10 client.?.objecter ms_handle_connect 0x7f095c13f710 2024-03-25T19:47:11.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.799+0000 7f0951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:11.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.799+0000 7f0951ffb640 10 client.?.objecter ms_handle_connect 0x7f095c1430c0 2024-03-25T19:47:11.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.799+0000 7f0951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:11.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.809+0000 7f0951ffb640 10 client.?.objecter ms_handle_connect 0x7f095c0a98f0 2024-03-25T19:47:11.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.809+0000 7f0951ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:11.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278768053 0 0) 0x7f095c1d8210 con 0x7f095c13f710 2024-03-25T19:47:11.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:11.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:11.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:11.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T19:47:11.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.813+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0940004eb0 con 0x7f095c13f710 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63887126 0 0) 0x7f095c1397c0 con 0x7f095c0a98f0 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 10 monclient(hunting): my global_id is 4479 2024-03-25T19:47:11.820 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.819+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c1d8210 con 0x7f095c0a98f0 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678729029 0 0) 0x7f095c1daa20 con 0x7f095c1430c0 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 10 monclient(hunting): my global_id is 4474 2024-03-25T19:47:11.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.821+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f095c1397c0 con 0x7f095c1430c0 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2803698021 0 0) 0x7f0940004eb0 con 0x7f095c13f710 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095c1daa20 con 0x7f095c13f710 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f094c001bc0 con 0x7f095c13f710 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:11.828 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.827+0000 7f0951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 97300950 0 0) 0x7f095c1d8210 con 0x7f095c0a98f0 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0940004eb0 con 0x7f095c0a98f0 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f09580019f0 con 0x7f095c0a98f0 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:11.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 375001490 0 0) 0x7f095c1397c0 con 0x7f095c1430c0 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095c1d8210 con 0x7f095c1430c0 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0954001a60 con 0x7f095c1430c0 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:11.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.831+0000 7f0951ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:11.834 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4199095316 0 0) 0x7f0940004eb0 con 0x7f095c0a98f0 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 >> v1:172.21.15.130:6790/0 conn(0x7f095c13f710 legacy=0x7f095c1d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 >> v1:172.21.15.163:6789/0 conn(0x7f095c1430c0 legacy=0x7f095c1d7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 1 monclient: found mon.a 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f095c1db650 con 0x7f095c0a98f0 2024-03-25T19:47:11.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f0951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 5 monclient: authenticate success, global_id 4479 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 10 monclient: _renew_subs 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095c137720 con 0x7f095c0a98f0 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 10 client.4479.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 10 monclient: _renew_subs 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f095c137d50 con 0x7f095c0a98f0 2024-03-25T19:47:11.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.842+0000 7f09638e6640 1 librados: init done 2024-03-25T19:47:11.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.843+0000 7f09638e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:11.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.843+0000 7f09638e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:11.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.843+0000 7f09638e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:11.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:11.843+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f095c1db840 con 0x7f095c0a98f0 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09580035f0 con 0x7f095c0a98f0 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0958003760 con 0x7f095c0a98f0 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f0951ffb640 10 monclient: dump: 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:12.165 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:12.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.164+0000 7f09537fe640 4 set_mon_vals no callback set 2024-03-25T19:47:12.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.165+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f0958019730 con 0x7f095c0a98f0 2024-03-25T19:47:12.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.165+0000 7f0960e81640 10 monclient: get_auth_request con 0x7f094006e110 auth_method 0 2024-03-25T19:47:12.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.166+0000 7f0951ffb640 10 client.4479.objecter ms_handle_connect 0x7f094006e110 2024-03-25T19:47:12.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f0958053420 con 0x7f095c0a98f0 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 10 client.4479.objecter ms_dispatch 0x7f095c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 3 client.4479.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 3 client.4479.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 20 client.4479.objecter dump_active .. 0 homeless 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f095804ed60 con 0x7f095c0a98f0 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:12.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.171+0000 7f0951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:12.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.391+0000 7f09638e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.391+0000 7f09638e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.391+0000 7f09638e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:12.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.391+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f095c074cc0 con 0x7f095c0a98f0 2024-03-25T19:47:12.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.419+0000 7f0951ffb640 1 -- 172.21.15.130:0/4158397285 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f0958050e60 con 0x7f095c0a98f0 2024-03-25T19:47:12.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.419+0000 7f0951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:12.420 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.419+0000 7f0951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:12.420 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:12.421 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:12.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.421+0000 7f09638e6640 10 librados: watch_flush enter 2024-03-25T19:47:12.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.421+0000 7f09638e6640 10 librados: watch_flush exit 2024-03-25T19:47:12.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.421+0000 7f09638e6640 20 client.4479.objecter shutdown clearing up homeless session... 2024-03-25T19:47:12.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.421+0000 7f09638e6640 10 client.4479.objecter successfully canceled tick 2024-03-25T19:47:12.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 >> v1:172.21.15.163:6800/1217380380 conn(0x7f094006e110 legacy=0x7f09400705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 10 monclient: shutdown 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 >> v1:172.21.15.130:6789/0 conn(0x7f095c0a98f0 legacy=0x7f095c1daf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 shutdown_connections 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 >> 172.21.15.130:0/4158397285 conn(0x7f095c0b7ea0 msgr2=0x7f095c13c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 shutdown_connections 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 -- 172.21.15.130:0/4158397285 wait complete. 2024-03-25T19:47:12.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:12.422+0000 7f09638e6640 1 librados: shutdown 2024-03-25T19:47:12.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:19.242 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:19.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.360+0000 7ff131ae4640 10 monclient: get_monmap_and_config 2024-03-25T19:47:19.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.360+0000 7ff131ae4640 10 monclient: build_initial_monmap 2024-03-25T19:47:19.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.360+0000 7ff131ae4640 10 monclient: monmap: 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:19.360937+0000 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:19.360937+0000 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:19.360 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.360+0000 7ff131ae4640 1 Processor -- start 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.360+0000 7ff131ae4640 1 -- start start 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient: init 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:19.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-c con 0x7ff12c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-b con 0x7ff12c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c142520 con 0x7ff12c0a98f0 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c0b28a0 con 0x7ff12c13f710 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c0b2a30 con 0x7ff12c1430c0 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:19.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.361+0000 7ff131ae4640 10 monclient(hunting): authenticate will time out at 5315.507443s 2024-03-25T19:47:19.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.365+0000 7ff12b7fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ff12c0a98f0 0x7ff12c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:45850/0 (socket says 172.21.15.130:45850) 2024-03-25T19:47:19.365 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.366+0000 7ff12b7fe640 1 -- 172.21.15.130:0/1242198850 learned_addr learned my addr 172.21.15.130:0/1242198850 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425918383 0 0) 0x7ff12c142520 con 0x7ff12c0a98f0 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T19:47:19.372 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.372+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff114003520 con 0x7ff12c0a98f0 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858200037 0 0) 0x7ff12c0b28a0 con 0x7ff12c13f710 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4477 2024-03-25T19:47:19.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.382+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c142520 con 0x7ff12c13f710 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3290029501 0 0) 0x7ff12c0b2a30 con 0x7ff12c1430c0 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T19:47:19.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.383+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c0b28a0 con 0x7ff12c1430c0 2024-03-25T19:47:19.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3770330271 0 0) 0x7ff114003520 con 0x7ff12c0a98f0 2024-03-25T19:47:19.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 >> v1:172.21.15.130:6790/0 conn(0x7ff12c1430c0 legacy=0x7ff12c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 >> v1:172.21.15.163:6789/0 conn(0x7ff12c13f710 legacy=0x7ff12c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 monclient: found mon.noname-a 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0b44c0 con 0x7ff12c0a98f0 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff11c001690 con 0x7ff12c0a98f0 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff131ae4640 5 monclient: authenticate success, global_id 4482 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff131ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:19.385 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff12a7fc640 10 monclient: dump: 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.384+0000 7ff131ae4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff11c0017f0 con 0x7ff12c0a98f0 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 1 -- 172.21.15.130:0/1242198850 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff11c001c00 con 0x7ff12c0a98f0 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff131ae4640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff131ae4640 4 set_mon_vals no callback set 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.399+0000 7ff12a7fc640 10 monclient: dump: 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.399 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:19.400 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:19.400 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:19.400 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 10 monclient: shutdown 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 1 -- 172.21.15.130:0/1242198850 >> v1:172.21.15.130:6789/0 conn(0x7ff12c0a98f0 legacy=0x7ff12c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 1 -- 172.21.15.130:0/1242198850 shutdown_connections 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 1 -- 172.21.15.130:0/1242198850 >> 172.21.15.130:0/1242198850 conn(0x7ff12c0b7ea0 msgr2=0x7ff12c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 1 -- 172.21.15.130:0/1242198850 shutdown_connections 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.400+0000 7ff131ae4640 1 -- 172.21.15.130:0/1242198850 wait complete. 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.401+0000 7ff131ae4640 10 monclient: build_initial_monmap 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.401+0000 7ff131ae4640 10 monclient: monmap: 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:19.401 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:19.402 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:19.402 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:19.402 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:19.402 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.401+0000 7ff131ae4640 1 librados: starting msgr at 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.401+0000 7ff131ae4640 1 librados: starting objecter 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 1 Processor -- start 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 1 -- start start 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 1 librados: setting wanted keys 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 1 librados: calling monclient init 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 10 monclient: init 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:19.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.402+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-b con 0x7ff12c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 10 monclient(hunting): picked mon.noname-c con 0x7ff12c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c1db050 con 0x7ff12c0a98f0 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c1dab70 con 0x7ff12c13f710 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff12c1dad00 con 0x7ff12c1430c0 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:19.404 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.403+0000 7ff131ae4640 10 monclient(hunting): authenticate will time out at 5315.548797s 2024-03-25T19:47:19.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.412+0000 7ff12affd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ff12c13f710 0x7ff12c1d7670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:46132/0 (socket says 172.21.15.130:46132) 2024-03-25T19:47:19.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.412+0000 7ff12affd640 1 -- 172.21.15.130:0/3711918041 learned_addr learned my addr 172.21.15.130:0/3711918041 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:19.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.413+0000 7ff113fff640 10 client.?.objecter ms_handle_connect 0x7ff12c1430c0 2024-03-25T19:47:19.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.413+0000 7ff113fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.414+0000 7ff113fff640 10 client.?.objecter ms_handle_connect 0x7ff12c13f710 2024-03-25T19:47:19.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.414+0000 7ff113fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3119241667 0 0) 0x7ff12c1dad00 con 0x7ff12c1430c0 2024-03-25T19:47:19.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 10 monclient(hunting): my global_id is 4478 2024-03-25T19:47:19.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.422+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff118004eb0 con 0x7ff12c1430c0 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539131744 0 0) 0x7ff12c1dab70 con 0x7ff12c13f710 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 10 monclient(hunting): my global_id is 4480 2024-03-25T19:47:19.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.424+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c1dad00 con 0x7ff12c13f710 2024-03-25T19:47:19.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.425+0000 7ff113fff640 10 client.?.objecter ms_handle_connect 0x7ff12c0a98f0 2024-03-25T19:47:19.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.425+0000 7ff113fff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:19.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.438+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2014586784 0 0) 0x7ff118004eb0 con 0x7ff12c1430c0 2024-03-25T19:47:19.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff12c1dab70 con 0x7ff12c1430c0 2024-03-25T19:47:19.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff124001bd0 con 0x7ff12c1430c0 2024-03-25T19:47:19.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 10 monclient(hunting): dump: 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.439 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.439+0000 7ff113fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.444+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784859462 0 0) 0x7ff12c1dad00 con 0x7ff12c13f710 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.444+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff118004eb0 con 0x7ff12c13f710 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.444+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff1200019f0 con 0x7ff12c13f710 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.444+0000 7ff113fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): dump: 2024-03-25T19:47:19.444 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:19.445 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760908624 0 0) 0x7ff12c1db050 con 0x7ff12c0a98f0 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 10 monclient(hunting): my global_id is 4485 2024-03-25T19:47:19.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.445+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff12c1dad00 con 0x7ff12c0a98f0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3505381636 0 0) 0x7ff12c1dab70 con 0x7ff12c1430c0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 >> v1:172.21.15.163:6789/0 conn(0x7ff12c13f710 legacy=0x7ff12c1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 >> v1:172.21.15.130:6789/0 conn(0x7ff12c0a98f0 legacy=0x7ff12c1db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 1 monclient: found mon.c 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff12c0b70a0 con 0x7ff12c1430c0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 10 monclient: _finish_auth 0 2024-03-25T19:47:19.450 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff113fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 5 monclient: authenticate success, global_id 4478 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 10 monclient: _renew_subs 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff12c0b72f0 con 0x7ff12c1430c0 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 10 monclient: _renew_subs 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff12c1e32a0 con 0x7ff12c1430c0 2024-03-25T19:47:19.451 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.450+0000 7ff131ae4640 1 librados: init done 2024-03-25T19:47:19.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.451+0000 7ff131ae4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:19.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.451+0000 7ff131ae4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:19.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.451+0000 7ff131ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:19.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.451+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12c0627b0 con 0x7ff12c1430c0 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff124002060 con 0x7ff12c1430c0 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff124003b30 con 0x7ff12c1430c0 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 10 monclient: dump: 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:19.768 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff129ffb640 4 set_mon_vals no callback set 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.768+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ff124019b40 con 0x7ff12c1430c0 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.769+0000 7ff12b7fe640 10 monclient: get_auth_request con 0x7ff11806df00 auth_method 0 2024-03-25T19:47:19.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.769+0000 7ff113fff640 10 client.4478.objecter ms_handle_connect 0x7ff11806df00 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ff1240538b0 con 0x7ff12c1430c0 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 10 client.4478.objecter ms_dispatch 0x7ff12c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff12404f1f0 con 0x7ff12c1430c0 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:19.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:19.775+0000 7ff113fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:20.028 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.028+0000 7ff131ae4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.028 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.028+0000 7ff131ae4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.028 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.028+0000 7ff131ae4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:20.028 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.028+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff12c074cc0 con 0x7ff12c1430c0 2024-03-25T19:47:20.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.207+0000 7ff113fff640 1 -- 172.21.15.130:0/3711918041 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ff1240512f0 con 0x7ff12c1430c0 2024-03-25T19:47:20.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.207+0000 7ff113fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:20.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.207+0000 7ff113fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:20.208 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:20.208 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 10 librados: watch_flush enter 2024-03-25T19:47:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 10 librados: watch_flush exit 2024-03-25T19:47:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-25T19:47:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 10 client.4478.objecter successfully canceled tick 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 >> v1:172.21.15.163:6800/1217380380 conn(0x7ff11806df00 legacy=0x7ff1180703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 10 monclient: shutdown 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 >> v1:172.21.15.130:6790/0 conn(0x7ff12c1430c0 legacy=0x7ff12c1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 shutdown_connections 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 >> 172.21.15.130:0/3711918041 conn(0x7ff12c0b7ea0 msgr2=0x7ff12c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 shutdown_connections 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 -- 172.21.15.130:0/3711918041 wait complete. 2024-03-25T19:47:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:20.209+0000 7ff131ae4640 1 librados: shutdown 2024-03-25T19:47:20.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:25.314 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:10.247 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.360 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:10.260 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.361 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:10.262 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.361 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:10.294 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.399 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:10.300 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.413 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:10.346 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.417 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:10.350 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.425 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:10.358 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:25.451 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:10.384 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:27.029 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.146+0000 7fa10296d640 10 monclient: get_monmap_and_config 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.146+0000 7fa10296d640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.146+0000 7fa10296d640 10 monclient: monmap: 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:27.147352+0000 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:27.147352+0000 2024-03-25T19:47:27.146 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.147 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:27.147 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:27.147 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:27.147 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:27.147 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.147+0000 7fa10296d640 1 Processor -- start 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.147+0000 7fa10296d640 1 -- start start 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.147+0000 7fa10296d640 10 monclient: init 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0fc0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0fc13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0fc1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc142520 con 0x7fa0fc1430c0 2024-03-25T19:47:27.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc0b28a0 con 0x7fa0fc13f710 2024-03-25T19:47:27.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc0b2a30 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.148+0000 7fa10296d640 10 monclient(hunting): authenticate will time out at 5323.293992s 2024-03-25T19:47:27.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.155+0000 7fa0fbfff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fa0fc0a98f0 0x7fa0fc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:38392/0 (socket says 172.21.15.130:38392) 2024-03-25T19:47:27.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.155+0000 7fa0fbfff640 1 -- 172.21.15.130:0/884797300 learned_addr learned my addr 172.21.15.130:0/884797300 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 390983005 0 0) 0x7fa0fc0b2a30 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 10 monclient(hunting): my global_id is 4481 2024-03-25T19:47:27.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.168+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e4003520 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502094767 0 0) 0x7fa0fc0b28a0 con 0x7fa0fc13f710 2024-03-25T19:47:27.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 10 monclient(hunting): my global_id is 4483 2024-03-25T19:47:27.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.175+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc0b2a30 con 0x7fa0fc13f710 2024-03-25T19:47:27.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.184+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531880310 0 0) 0x7fa0fc142520 con 0x7fa0fc1430c0 2024-03-25T19:47:27.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.184+0000 7fa0faffd640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.184+0000 7fa0faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.184+0000 7fa0faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.184+0000 7fa0faffd640 10 monclient(hunting): my global_id is 4488 2024-03-25T19:47:27.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.185+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc0b28a0 con 0x7fa0fc1430c0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 129461218 0 0) 0x7fa0e4003520 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 >> v1:172.21.15.163:6789/0 conn(0x7fa0fc13f710 legacy=0x7fa0fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 >> v1:172.21.15.130:6789/0 conn(0x7fa0fc1430c0 legacy=0x7fa0fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 monclient: found mon.noname-b 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0b44c0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0ec001690 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa0faffd640 10 monclient: dump: 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.192 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa10296d640 5 monclient: authenticate success, global_id 4481 2024-03-25T19:47:27.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.190+0000 7fa10296d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec0017f0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 1 -- 172.21.15.130:0/884797300 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0ec001c00 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa10296d640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa10296d640 4 set_mon_vals no callback set 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:27.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.222+0000 7fa0faffd640 10 monclient: dump: 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:27.223 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 10 monclient: shutdown 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 1 -- 172.21.15.130:0/884797300 >> v1:172.21.15.130:6790/0 conn(0x7fa0fc0a98f0 legacy=0x7fa0fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 1 -- 172.21.15.130:0/884797300 shutdown_connections 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 1 -- 172.21.15.130:0/884797300 >> 172.21.15.130:0/884797300 conn(0x7fa0fc0b7ea0 msgr2=0x7fa0fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 1 -- 172.21.15.130:0/884797300 shutdown_connections 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.223+0000 7fa10296d640 1 -- 172.21.15.130:0/884797300 wait complete. 2024-03-25T19:47:27.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.224+0000 7fa10296d640 10 monclient: build_initial_monmap 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.224+0000 7fa10296d640 10 monclient: monmap: 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:27.227 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.224+0000 7fa10296d640 1 librados: starting msgr at 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.224+0000 7fa10296d640 1 librados: starting objecter 2024-03-25T19:47:27.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.224+0000 7fa10296d640 1 Processor -- start 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 -- start start 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 librados: setting wanted keys 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 librados: calling monclient init 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient: init 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-a con 0x7fa0fc0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-b con 0x7fa0fc13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient(hunting): picked mon.noname-c con 0x7fa0fc1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc13be70 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc131660 con 0x7fa0fc13f710 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc1317f0 con 0x7fa0fc1430c0 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:27.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.225+0000 7fa10296d640 10 monclient(hunting): authenticate will time out at 5323.371498s 2024-03-25T19:47:27.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.232+0000 7fa0fbfff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fa0fc0a98f0 0x7fa0fc130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:33272/0 (socket says 172.21.15.130:33272) 2024-03-25T19:47:27.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.232+0000 7fa0fbfff640 1 -- 172.21.15.130:0/1453802421 learned_addr learned my addr 172.21.15.130:0/1453802421 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:27.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.234+0000 7fa0f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0fc0a98f0 2024-03-25T19:47:27.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.234+0000 7fa0f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.235+0000 7fa0f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0fc13f710 2024-03-25T19:47:27.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.235+0000 7fa0f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837866073 0 0) 0x7fa0fc13be70 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 10 monclient(hunting): my global_id is 4491 2024-03-25T19:47:27.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.246+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8004eb0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.252 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1991068045 0 0) 0x7fa0fc131660 con 0x7fa0fc13f710 2024-03-25T19:47:27.252 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:47:27.252 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:27.252 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:27.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 10 monclient(hunting): my global_id is 4486 2024-03-25T19:47:27.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.253+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc13be70 con 0x7fa0fc13f710 2024-03-25T19:47:27.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.255+0000 7fa0f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa0fc1430c0 2024-03-25T19:47:27.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.255+0000 7fa0f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3015302366 0 0) 0x7fa0e8004eb0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0fc131660 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0ec001bc0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 10 monclient(hunting): dump: 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:27.257 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:27.258 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:27.258 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:27.258 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.257+0000 7fa0f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.265+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1548232 0 0) 0x7fa0fc131660 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 >> v1:172.21.15.130:6790/0 conn(0x7fa0fc1430c0 legacy=0x7fa0fc1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 >> v1:172.21.15.163:6789/0 conn(0x7fa0fc13f710 legacy=0x7fa0fc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 1 monclient: found mon.a 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc131980 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa0f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:27.266 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 5 monclient: authenticate success, global_id 4491 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 10 monclient: _renew_subs 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0fc135c60 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 10 monclient: _renew_subs 2024-03-25T19:47:27.267 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:27.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc0b7770 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.266+0000 7fa10296d640 1 librados: init done 2024-03-25T19:47:27.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.267+0000 7fa10296d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.267+0000 7fa10296d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.267+0000 7fa10296d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:27.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.267+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc1371d0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec004b00 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:27.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0ec004f10 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0f8ff9640 10 monclient: dump: 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:27.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.579+0000 7fa0fa7fc640 4 set_mon_vals no callback set 2024-03-25T19:47:27.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.580+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fa0ec005130 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.581+0000 7fa0fb7fe640 10 monclient: get_auth_request con 0x7fa0e806de30 auth_method 0 2024-03-25T19:47:27.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.581+0000 7fa0f8ff9640 10 client.4491.objecter ms_handle_connect 0x7fa0e806de30 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fa0ec0011f0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 10 client.4491.objecter ms_dispatch 0x7fa0fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:27.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0ec050400 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:27.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.585+0000 7fa0f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:27.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.842+0000 7fa10296d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:27.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.842+0000 7fa10296d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:27.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.842+0000 7fa10296d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:27.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.842+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0fc0627b0 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.870+0000 7fa0f8ff9640 1 -- 172.21.15.130:0/1453802421 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fa0ec052330 con 0x7fa0fc0a98f0 2024-03-25T19:47:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.871+0000 7fa0f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:27.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.871+0000 7fa0f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:27.871 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:27.871 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:27.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 10 librados: watch_flush enter 2024-03-25T19:47:27.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 10 librados: watch_flush exit 2024-03-25T19:47:27.873 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 10 client.4491.objecter successfully canceled tick 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 >> v1:172.21.15.163:6800/1217380380 conn(0x7fa0e806de30 legacy=0x7fa0e80702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 10 monclient: shutdown 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 >> v1:172.21.15.130:6789/0 conn(0x7fa0fc0a98f0 legacy=0x7fa0fc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 shutdown_connections 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 >> 172.21.15.130:0/1453802421 conn(0x7fa0fc0b7ea0 msgr2=0x7fa0fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 shutdown_connections 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 -- 172.21.15.130:0/1453802421 wait complete. 2024-03-25T19:47:27.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:27.873+0000 7fa10296d640 1 librados: shutdown 2024-03-25T19:47:27.886 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:34.692 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:34.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.807+0000 7ffa3bcf4640 10 monclient: get_monmap_and_config 2024-03-25T19:47:34.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.807+0000 7ffa3bcf4640 10 monclient: build_initial_monmap 2024-03-25T19:47:34.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.807+0000 7ffa3bcf4640 10 monclient: monmap: 2024-03-25T19:47:34.807 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:34.808458+0000 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:34.808458+0000 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:34.808 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.808+0000 7ffa3bcf4640 1 Processor -- start 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.808+0000 7ffa3bcf4640 1 -- start start 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.808+0000 7ffa3bcf4640 10 monclient: init 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-b con 0x7ffa340a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:34.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-c con 0x7ffa3413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-a con 0x7ffa341430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa34142520 con 0x7ffa341430c0 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa340b28a0 con 0x7ffa3413f710 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa340b2a30 con 0x7ffa340a98f0 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:34.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.809+0000 7ffa3bcf4640 10 monclient(hunting): authenticate will time out at 5330.954945s 2024-03-25T19:47:34.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.815+0000 7ffa39a90640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7ffa340a98f0 0x7ffa340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:43402/0 (socket says 172.21.15.130:43402) 2024-03-25T19:47:34.815 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.815+0000 7ffa39a90640 1 -- 172.21.15.130:0/4217890550 learned_addr learned my addr 172.21.15.130:0/4217890550 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 508757961 0 0) 0x7ffa340b28a0 con 0x7ffa3413f710 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 10 monclient(hunting): my global_id is 4489 2024-03-25T19:47:34.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.826+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa20003520 con 0x7ffa3413f710 2024-03-25T19:47:34.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1883023771 0 0) 0x7ffa34142520 con 0x7ffa341430c0 2024-03-25T19:47:34.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:47:34.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:34.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:34.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 10 monclient(hunting): my global_id is 4494 2024-03-25T19:47:34.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.830+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa340b28a0 con 0x7ffa341430c0 2024-03-25T19:47:34.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361034650 0 0) 0x7ffa340b2a30 con 0x7ffa340a98f0 2024-03-25T19:47:34.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 10 monclient(hunting): _negotiate 2024-03-25T19:47:34.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:34.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:34.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 10 monclient(hunting): my global_id is 4487 2024-03-25T19:47:34.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.835+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa34142520 con 0x7ffa340a98f0 2024-03-25T19:47:34.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2989060122 0 0) 0x7ffa340b28a0 con 0x7ffa341430c0 2024-03-25T19:47:34.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 >> v1:172.21.15.130:6790/0 conn(0x7ffa340a98f0 legacy=0x7ffa340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:34.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 legacy=0x7ffa341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:34.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: _finish_hunting 0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 monclient: found mon.noname-a 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa340b44c0 con 0x7ffa341430c0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: _finish_auth 0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: _check_auth_tickets 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa2c0019f0 con 0x7ffa341430c0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa38a8e640 10 monclient: dump: 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:34.841 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa3bcf4640 5 monclient: authenticate success, global_id 4494 2024-03-25T19:47:34.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.840+0000 7ffa3bcf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa2c001b50 con 0x7ffa341430c0 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 1 -- 172.21.15.130:0/4217890550 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa2c001f60 con 0x7ffa341430c0 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa3bcf4640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa3bcf4640 4 set_mon_vals no callback set 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.844+0000 7ffa38a8e640 10 monclient: dump: 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.844 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:34.845 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 10 monclient: shutdown 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/4217890550 >> v1:172.21.15.130:6789/0 conn(0x7ffa341430c0 legacy=0x7ffa3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/4217890550 shutdown_connections 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/4217890550 >> 172.21.15.130:0/4217890550 conn(0x7ffa340b7ea0 msgr2=0x7ffa340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/4217890550 shutdown_connections 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.845+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/4217890550 wait complete. 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient: build_initial_monmap 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient: monmap: 2024-03-25T19:47:34.847 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 librados: starting msgr at 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 librados: starting objecter 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 Processor -- start 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 -- start start 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 librados: setting wanted keys 2024-03-25T19:47:34.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 1 librados: calling monclient init 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient: init 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-c con 0x7ffa340a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.846+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-b con 0x7ffa3413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 10 monclient(hunting): picked mon.noname-a con 0x7ffa341430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa34135bf0 con 0x7ffa341430c0 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa340b6d10 con 0x7ffa3413f710 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa341317f0 con 0x7ffa340a98f0 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:34.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.847+0000 7ffa3bcf4640 10 monclient(hunting): authenticate will time out at 5330.992699s 2024-03-25T19:47:34.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.851+0000 7ffa3a291640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ffa341430c0 0x7ffa341db370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:38086/0 (socket says 172.21.15.130:38086) 2024-03-25T19:47:34.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.851+0000 7ffa3a291640 1 -- 172.21.15.130:0/2085836554 learned_addr learned my addr 172.21.15.130:0/2085836554 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:34.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.853+0000 7ffa2a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa341430c0 2024-03-25T19:47:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.853+0000 7ffa2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:34.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.855+0000 7ffa2a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa340a98f0 2024-03-25T19:47:34.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.855+0000 7ffa2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082246749 0 0) 0x7ffa34135bf0 con 0x7ffa341430c0 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 10 monclient(hunting): my global_id is 4497 2024-03-25T19:47:34.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.856+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa18004eb0 con 0x7ffa341430c0 2024-03-25T19:47:34.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.860+0000 7ffa2a7fc640 10 client.?.objecter ms_handle_connect 0x7ffa3413f710 2024-03-25T19:47:34.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.860+0000 7ffa2a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:34.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.861+0000 7ffa3928f640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 legacy=0x7ffa341dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T19:47:34.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.861+0000 7ffa3928f640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 legacy=0x7ffa341dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:47:34.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.861+0000 7ffa3928f640 1 --1- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 0x7ffa341dac60 :-1 s=OPENED pgs=41561 cs=1 l=1).handle_message read tag failed 2024-03-25T19:47:34.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.861+0000 7ffa3928f640 1 --1- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 0x7ffa341dac60 :-1 s=OPENED pgs=41561 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:47:34.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.861+0000 7ffa2a7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.163:6789/0 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968886833 0 0) 0x7ffa341317f0 con 0x7ffa340a98f0 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 10 monclient(hunting): my global_id is 4490 2024-03-25T19:47:34.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.863+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa34135bf0 con 0x7ffa340a98f0 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3422200805 0 0) 0x7ffa18004eb0 con 0x7ffa341430c0 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffa341317f0 con 0x7ffa341430c0 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa2c001fa0 con 0x7ffa341430c0 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 10 monclient(hunting): dump: 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:34.867 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:34.868 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:34.868 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:34.868 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:34.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.867+0000 7ffa2a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 537216978 0 0) 0x7ffa341317f0 con 0x7ffa341430c0 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.130:6790/0 conn(0x7ffa340a98f0 legacy=0x7ffa341310e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6789/0 conn(0x7ffa3413f710 legacy=0x7ffa341dac60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 1 monclient: found mon.a 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa34131980 con 0x7ffa341430c0 2024-03-25T19:47:34.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa2a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa3bcf4640 5 monclient: authenticate success, global_id 4497 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa3bcf4640 10 monclient: _renew_subs 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa3bcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.880+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa340b76e0 con 0x7ffa341430c0 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.881+0000 7ffa3bcf4640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.881+0000 7ffa3bcf4640 10 monclient: _renew_subs 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.881+0000 7ffa3bcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.881+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa34135270 con 0x7ffa341430c0 2024-03-25T19:47:34.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.881+0000 7ffa3bcf4640 1 librados: init done 2024-03-25T19:47:34.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.882+0000 7ffa3bcf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:34.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.882+0000 7ffa3bcf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:34.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.882+0000 7ffa3bcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:34.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:34.882+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa340627b0 con 0x7ffa341430c0 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffa2c003c20 con 0x7ffa341430c0 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffa2c003f70 con 0x7ffa341430c0 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2a7fc640 10 monclient: dump: 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:35.197 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:35.198 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.197+0000 7ffa2bfff640 4 set_mon_vals no callback set 2024-03-25T19:47:35.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.198+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ffa2c004190 con 0x7ffa341430c0 2024-03-25T19:47:35.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.199+0000 7ffa39a90640 10 monclient: get_auth_request con 0x7ffa1806dd00 auth_method 0 2024-03-25T19:47:35.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.199+0000 7ffa2a7fc640 10 client.4497.objecter ms_handle_connect 0x7ffa1806dd00 2024-03-25T19:47:35.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ffa2c054b60 con 0x7ffa341430c0 2024-03-25T19:47:35.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 10 client.4497.objecter ms_dispatch 0x7ffa34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:35.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:35.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:35.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T19:47:35.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.203+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffa2c050670 con 0x7ffa341430c0 2024-03-25T19:47:35.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.204+0000 7ffa2a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:35.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.204+0000 7ffa2a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:35.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.433+0000 7ffa3bcf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.433+0000 7ffa3bcf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.433+0000 7ffa3bcf4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:35.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.433+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffa34135da0 con 0x7ffa341430c0 2024-03-25T19:47:35.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.463+0000 7ffa2a7fc640 1 -- 172.21.15.130:0/2085836554 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ffa2c0525a0 con 0x7ffa341430c0 2024-03-25T19:47:35.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.463+0000 7ffa2a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:35.463 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.463+0000 7ffa2a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:35.463 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:35.463 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:35.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.465+0000 7ffa3bcf4640 10 librados: watch_flush enter 2024-03-25T19:47:35.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.465+0000 7ffa3bcf4640 10 librados: watch_flush exit 2024-03-25T19:47:35.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 10 client.4497.objecter successfully canceled tick 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.163:6800/1217380380 conn(0x7ffa1806dd00 legacy=0x7ffa180701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 10 monclient: shutdown 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 >> v1:172.21.15.130:6789/0 conn(0x7ffa341430c0 legacy=0x7ffa341db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 shutdown_connections 2024-03-25T19:47:35.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 >> 172.21.15.130:0/2085836554 conn(0x7ffa340b7ea0 msgr2=0x7ffa34074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:35.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 shutdown_connections 2024-03-25T19:47:35.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 -- 172.21.15.130:0/2085836554 wait complete. 2024-03-25T19:47:35.467 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:35.466+0000 7ffa3bcf4640 1 librados: shutdown 2024-03-25T19:47:35.479 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:42.285 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.403+0000 7fabe7d15640 10 monclient: get_monmap_and_config 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.403+0000 7fabe7d15640 10 monclient: build_initial_monmap 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.403+0000 7fabe7d15640 10 monclient: monmap: 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:42.404174+0000 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:42.404174+0000 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:42.403 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:42.404 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 1 Processor -- start 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 1 -- start start 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 10 monclient: init 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-c con 0x7fabe00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.404+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-b con 0x7fabe013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-a con 0x7fabe01430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:42.405 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0142520 con 0x7fabe01430c0 2024-03-25T19:47:42.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b28a0 con 0x7fabe00a98f0 2024-03-25T19:47:42.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe00b2a30 con 0x7fabe013f710 2024-03-25T19:47:42.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:42.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.405+0000 7fabe7d15640 10 monclient(hunting): authenticate will time out at 5338.550775s 2024-03-25T19:47:42.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.408+0000 7fabe5ab1640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fabe00a98f0 0x7fabe00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:57372/0 (socket says 172.21.15.130:57372) 2024-03-25T19:47:42.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.408+0000 7fabe5ab1640 1 -- 172.21.15.130:0/3838295773 learned_addr learned my addr 172.21.15.130:0/3838295773 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4015334001 0 0) 0x7fabe00b28a0 con 0x7fabe00a98f0 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 10 monclient(hunting): my global_id is 4495 2024-03-25T19:47:42.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.417+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabcc003520 con 0x7fabe00a98f0 2024-03-25T19:47:42.422 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656374153 0 0) 0x7fabe00b2a30 con 0x7fabe013f710 2024-03-25T19:47:42.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 10 monclient(hunting): my global_id is 4493 2024-03-25T19:47:42.423 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.423+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe00b28a0 con 0x7fabe013f710 2024-03-25T19:47:42.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316510324 0 0) 0x7fabe0142520 con 0x7fabe01430c0 2024-03-25T19:47:42.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 10 monclient(hunting): my global_id is 4500 2024-03-25T19:47:42.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.429+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe00b2a30 con 0x7fabe01430c0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680002275 0 0) 0x7fabcc003520 con 0x7fabe00a98f0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 >> v1:172.21.15.130:6790/0 conn(0x7fabe013f710 legacy=0x7fabe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 >> v1:172.21.15.130:6789/0 conn(0x7fabe01430c0 legacy=0x7fabe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: _finish_hunting 0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 monclient: found mon.noname-c 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00b44c0 con 0x7fabe00a98f0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: _finish_auth 0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: _check_auth_tickets 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabdc001690 con 0x7fabe00a98f0 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:42.431 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe4aaf640 10 monclient: dump: 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe7d15640 5 monclient: authenticate success, global_id 4495 2024-03-25T19:47:42.432 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.430+0000 7fabe7d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabdc0017f0 con 0x7fabe00a98f0 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 1 -- 172.21.15.130:0/3838295773 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabdc001c00 con 0x7fabe00a98f0 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe7d15640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe7d15640 4 set_mon_vals no callback set 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.442+0000 7fabe4aaf640 10 monclient: dump: 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.442 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.443 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.443 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.443 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.443 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.443 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 10 monclient: shutdown 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 1 -- 172.21.15.130:0/3838295773 >> v1:172.21.15.163:6789/0 conn(0x7fabe00a98f0 legacy=0x7fabe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 1 -- 172.21.15.130:0/3838295773 shutdown_connections 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 1 -- 172.21.15.130:0/3838295773 >> 172.21.15.130:0/3838295773 conn(0x7fabe00b7ea0 msgr2=0x7fabe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 1 -- 172.21.15.130:0/3838295773 shutdown_connections 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.443+0000 7fabe7d15640 1 -- 172.21.15.130:0/3838295773 wait complete. 2024-03-25T19:47:42.444 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 10 monclient: build_initial_monmap 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 10 monclient: monmap: 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 librados: starting msgr at 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 librados: starting objecter 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 Processor -- start 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 -- start start 2024-03-25T19:47:42.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 librados: setting wanted keys 2024-03-25T19:47:42.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 1 librados: calling monclient init 2024-03-25T19:47:42.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.444+0000 7fabe7d15640 10 monclient: init 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-b con 0x7fabe00a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-c con 0x7fabe013f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient(hunting): picked mon.noname-a con 0x7fabe01430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0138250 con 0x7fabe01430c0 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0137d50 con 0x7fabe00a98f0 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabe0137ee0 con 0x7fabe013f710 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:42.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.445+0000 7fabe7d15640 10 monclient(hunting): authenticate will time out at 5338.591212s 2024-03-25T19:47:42.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.447+0000 7fabe62b2640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fabe01430c0 0x7fabe01d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:58102/0 (socket says 172.21.15.130:58102) 2024-03-25T19:47:42.448 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.447+0000 7fabe62b2640 1 -- 172.21.15.130:0/413726775 learned_addr learned my addr 172.21.15.130:0/413726775 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:42.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.449+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe01430c0 2024-03-25T19:47:42.449 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.449+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.452+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe013f710 2024-03-25T19:47:42.452 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.452+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148968906 0 0) 0x7fabe0138250 con 0x7fabe01430c0 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4503 2024-03-25T19:47:42.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.455+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabc4004eb0 con 0x7fabe01430c0 2024-03-25T19:47:42.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.466+0000 7fabd67fc640 10 client.?.objecter ms_handle_connect 0x7fabe00a98f0 2024-03-25T19:47:42.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.466+0000 7fabd67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:42.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659281710 0 0) 0x7fabe0137ee0 con 0x7fabe013f710 2024-03-25T19:47:42.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4496 2024-03-25T19:47:42.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.471+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0138250 con 0x7fabe013f710 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51286121 0 0) 0x7fabe0137d50 con 0x7fabe00a98f0 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 10 monclient(hunting): my global_id is 4498 2024-03-25T19:47:42.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.474+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabe0137ee0 con 0x7fabe00a98f0 2024-03-25T19:47:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.479+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1025254629 0 0) 0x7fabe0138250 con 0x7fabe013f710 2024-03-25T19:47:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe0137d50 con 0x7fabe013f710 2024-03-25T19:47:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabd0001970 con 0x7fabe013f710 2024-03-25T19:47:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 10 monclient(hunting): dump: 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.480+0000 7fabd67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2321870322 0 0) 0x7fabe0137ee0 con 0x7fabe00a98f0 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe0138250 con 0x7fabe00a98f0 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabdc001b00 con 0x7fabe00a98f0 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.484+0000 7fabd67fc640 10 monclient(hunting): dump: 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.484 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.485 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.485 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 117794689 0 0) 0x7fabc4004eb0 con 0x7fabe01430c0 2024-03-25T19:47:42.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fabe0137ee0 con 0x7fabe01430c0 2024-03-25T19:47:42.485 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabd8001c70 con 0x7fabe01430c0 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.486+0000 7fabd67fc640 10 monclient(hunting): dump: 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.486 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3190196587 0 0) 0x7fabe0137d50 con 0x7fabe013f710 2024-03-25T19:47:42.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 >> v1:172.21.15.163:6789/0 conn(0x7fabe00a98f0 legacy=0x7fabe0137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 >> v1:172.21.15.130:6789/0 conn(0x7fabe01430c0 legacy=0x7fabe01d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:42.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:47:42.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 1 monclient: found mon.c 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00b7730 con 0x7fabe013f710 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 10 monclient: _finish_auth 0 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 5 monclient: authenticate success, global_id 4496 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 10 monclient: _renew_subs 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe00b7920 con 0x7fabe013f710 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 10 client.4496.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 10 monclient: _renew_subs 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fabe01e33d0 con 0x7fabe013f710 2024-03-25T19:47:42.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.488+0000 7fabe7d15640 1 librados: init done 2024-03-25T19:47:42.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.489+0000 7fabe7d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:42.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.489+0000 7fabe7d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:42.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.489+0000 7fabe7d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:42.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.489+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabe00627b0 con 0x7fabe013f710 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabd00035f0 con 0x7fabe013f710 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fabd0003900 con 0x7fabe013f710 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd67fc640 10 monclient: dump: 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.785 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:42.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.785+0000 7fabd7fff640 4 set_mon_vals no callback set 2024-03-25T19:47:42.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.786+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fabd0019910 con 0x7fabe013f710 2024-03-25T19:47:42.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.786+0000 7fabe5ab1640 10 monclient: get_auth_request con 0x7fabc406e110 auth_method 0 2024-03-25T19:47:42.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.787+0000 7fabd67fc640 10 client.4496.objecter ms_handle_connect 0x7fabc406e110 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fabd0053580 con 0x7fabe013f710 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 10 client.4496.objecter ms_dispatch 0x7fabe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 3 client.4496.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 3 client.4496.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 20 client.4496.objecter dump_active .. 0 homeless 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fabd004eec0 con 0x7fabe013f710 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:42.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:42.791+0000 7fabd67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:43.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.046+0000 7fabe7d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.046+0000 7fabe7d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.046+0000 7fabe7d15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:43.046 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.046+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabe013df50 con 0x7fabe013f710 2024-03-25T19:47:43.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.065+0000 7fabd67fc640 1 -- 172.21.15.130:0/413726775 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fabd0050fc0 con 0x7fabe013f710 2024-03-25T19:47:43.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.065+0000 7fabd67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:43.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.065+0000 7fabd67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:43.065 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:43.066 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.068+0000 7fabe7d15640 10 librados: watch_flush enter 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.068+0000 7fabe7d15640 10 librados: watch_flush exit 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.068+0000 7fabe7d15640 20 client.4496.objecter shutdown clearing up homeless session... 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 10 client.4496.objecter successfully canceled tick 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 >> v1:172.21.15.163:6800/1217380380 conn(0x7fabc406e110 legacy=0x7fabc40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 10 monclient: shutdown 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 >> v1:172.21.15.130:6790/0 conn(0x7fabe013f710 legacy=0x7fabe01d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:43.069 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 shutdown_connections 2024-03-25T19:47:43.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 >> 172.21.15.130:0/413726775 conn(0x7fabe00b7ea0 msgr2=0x7fabe0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:43.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 shutdown_connections 2024-03-25T19:47:43.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 -- 172.21.15.130:0/413726775 wait complete. 2024-03-25T19:47:43.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:43.069+0000 7fabe7d15640 1 librados: shutdown 2024-03-25T19:47:43.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:45.467 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:30.400 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.513 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:30.414 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.514 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:30.447 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.515 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:30.416 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.527 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:30.428 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.549 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:30.482 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.605 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:30.537 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.609 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:30.542 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:45.646 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:30.579 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:47:49.887 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.005+0000 7f86ab9f6640 10 monclient: get_monmap_and_config 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.005+0000 7f86ab9f6640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.005+0000 7f86ab9f6640 10 monclient: monmap: 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:50.006281+0000 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:50.006281+0000 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:50.005 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:50.006 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:50.006 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:50.006 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.006+0000 7f86ab9f6640 1 Processor -- start 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.006+0000 7f86ab9f6640 1 -- start start 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.006+0000 7f86ab9f6640 10 monclient: init 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f86a40b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f86a40b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f86a414be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40af940 con 0x7f86a414be90 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b0a70 con 0x7f86a40b1730 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a40b0d50 con 0x7f86a40b4d60 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.007 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.007+0000 7f86ab9f6640 10 monclient(hunting): authenticate will time out at 5346.152777s 2024-03-25T19:47:50.014 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.014+0000 7f86a9f93640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f86a414be90 0x7f86a414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:35154/0 (socket says 172.21.15.130:35154) 2024-03-25T19:47:50.014 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.014+0000 7f86a9f93640 1 -- 172.21.15.130:0/3492951129 learned_addr learned my addr 172.21.15.130:0/3492951129 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:50.021 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617762289 0 0) 0x7f86a40af940 con 0x7f86a414be90 2024-03-25T19:47:50.021 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.021 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.021 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.021 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 10 monclient(hunting): my global_id is 4506 2024-03-25T19:47:50.022 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.022+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8690003520 con 0x7f86a414be90 2024-03-25T19:47:50.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208225076 0 0) 0x7f8690003520 con 0x7f86a414be90 2024-03-25T19:47:50.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 >> v1:172.21.15.130:6790/0 conn(0x7f86a40b4d60 legacy=0x7f86a413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 >> v1:172.21.15.163:6789/0 conn(0x7f86a40b1730 legacy=0x7f86a40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.032 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 monclient: found mon.noname-a 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a40b1030 con 0x7f86a414be90 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f869c001a10 con 0x7f86a414be90 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f869bfff640 10 monclient: dump: 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:50.033 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:50.034 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:50.034 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:50.034 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f86ab9f6640 5 monclient: authenticate success, global_id 4506 2024-03-25T19:47:50.034 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.032+0000 7f86ab9f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f869c001b70 con 0x7f86a414be90 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 1 -- 172.21.15.130:0/3492951129 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f869c001fc0 con 0x7f86a414be90 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f86ab9f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f86ab9f6640 4 set_mon_vals no callback set 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.052+0000 7f869bfff640 10 monclient: dump: 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:50.052 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:50.053 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 10 monclient: shutdown 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 1 -- 172.21.15.130:0/3492951129 >> v1:172.21.15.130:6789/0 conn(0x7f86a414be90 legacy=0x7f86a414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 1 -- 172.21.15.130:0/3492951129 shutdown_connections 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 1 -- 172.21.15.130:0/3492951129 >> 172.21.15.130:0/3492951129 conn(0x7f86a40617a0 msgr2=0x7f86a4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 1 -- 172.21.15.130:0/3492951129 shutdown_connections 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.053+0000 7f86ab9f6640 1 -- 172.21.15.130:0/3492951129 wait complete. 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 10 monclient: build_initial_monmap 2024-03-25T19:47:50.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 10 monclient: monmap: 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 librados: starting msgr at 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 librados: starting objecter 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 Processor -- start 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 -- start start 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 librados: setting wanted keys 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 1 librados: calling monclient init 2024-03-25T19:47:50.056 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.054+0000 7f86ab9f6640 10 monclient: init 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-c con 0x7f86a40b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-b con 0x7f86a40b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient(hunting): picked mon.noname-a con 0x7f86a414be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a413bd10 con 0x7f86a414be90 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a41dadb0 con 0x7f86a40b4d60 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86a4138950 con 0x7f86a40b1730 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:50.057 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.055+0000 7f86ab9f6640 10 monclient(hunting): authenticate will time out at 5346.200986s 2024-03-25T19:47:50.061 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.062+0000 7f86a9792640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f86a40b1730 0x7f86a4138240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:56216/0 (socket says 172.21.15.130:56216) 2024-03-25T19:47:50.061 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.062+0000 7f86a9792640 1 -- 172.21.15.130:0/466280511 learned_addr learned my addr 172.21.15.130:0/466280511 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:50.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.062+0000 7f8699ffb640 10 client.?.objecter ms_handle_connect 0x7f86a40b4d60 2024-03-25T19:47:50.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.062+0000 7f8699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.063 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.064+0000 7f8699ffb640 10 client.?.objecter ms_handle_connect 0x7f86a40b1730 2024-03-25T19:47:50.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.064+0000 7f8699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.071+0000 7f8699ffb640 10 client.?.objecter ms_handle_connect 0x7f86a414be90 2024-03-25T19:47:50.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.071+0000 7f8699ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711832207 0 0) 0x7f86a413bd10 con 0x7f86a414be90 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 10 monclient(hunting): my global_id is 4509 2024-03-25T19:47:50.077 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.077+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8688004eb0 con 0x7f86a414be90 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963440156 0 0) 0x7f8688004eb0 con 0x7f86a414be90 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f86a413bd10 con 0x7f86a414be90 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f869c001e80 con 0x7f86a414be90 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:47:50.090 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 10 monclient(hunting): dump: 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.091 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.090+0000 7f8699ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:50.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4071414713 0 0) 0x7f86a413bd10 con 0x7f86a414be90 2024-03-25T19:47:50.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 >> v1:172.21.15.130:6790/0 conn(0x7f86a40b1730 legacy=0x7f86a4138240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 >> v1:172.21.15.163:6789/0 conn(0x7f86a40b4d60 legacy=0x7f86a413a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:47:50.141 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 1 monclient: found mon.a 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a413ab60 con 0x7f86a414be90 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 10 monclient: _finish_auth 0 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f8699ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 5 monclient: authenticate success, global_id 4509 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 10 monclient: _renew_subs 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a41db9e0 con 0x7f86a414be90 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 10 monclient: _renew_subs 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.141+0000 7f86ab9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.142+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a4139890 con 0x7f86a414be90 2024-03-25T19:47:50.142 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.142+0000 7f86ab9f6640 1 librados: init done 2024-03-25T19:47:50.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.143+0000 7f86ab9f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.143+0000 7f86ab9f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.143+0000 7f86ab9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.143+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a413acf0 con 0x7f86a414be90 2024-03-25T19:47:50.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f869c004e10 con 0x7f86a414be90 2024-03-25T19:47:50.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:50.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f869c005120 con 0x7f86a414be90 2024-03-25T19:47:50.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:50.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f8699ffb640 10 monclient: dump: 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:50.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.434+0000 7f869b7fe640 4 set_mon_vals no callback set 2024-03-25T19:47:50.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.435+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f869c01b0f0 con 0x7f86a414be90 2024-03-25T19:47:50.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.436+0000 7f86a9792640 10 monclient: get_auth_request con 0x7f868806dc40 auth_method 0 2024-03-25T19:47:50.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.436+0000 7f8699ffb640 10 client.4509.objecter ms_handle_connect 0x7f868806dc40 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f869c054b20 con 0x7f86a414be90 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 10 client.4509.objecter ms_dispatch 0x7f86a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f869c050630 con 0x7f86a414be90 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:50.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.441+0000 7f8699ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:50.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.705+0000 7f86ab9f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:50.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.705+0000 7f86ab9f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:50.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.705+0000 7f86ab9f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:47:50.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.705+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86a40627b0 con 0x7f86a414be90 2024-03-25T19:47:50.726 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.726+0000 7f8699ffb640 1 -- 172.21.15.130:0/466280511 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f869c052560 con 0x7f86a414be90 2024-03-25T19:47:50.726 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.726+0000 7f8699ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:50.726 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.726+0000 7f8699ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:50.726 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:50.726 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:50.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.728+0000 7f86ab9f6640 10 librados: watch_flush enter 2024-03-25T19:47:50.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.728+0000 7f86ab9f6640 10 librados: watch_flush exit 2024-03-25T19:47:50.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.728+0000 7f86ab9f6640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T19:47:50.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.728+0000 7f86ab9f6640 10 client.4509.objecter successfully canceled tick 2024-03-25T19:47:50.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 >> v1:172.21.15.163:6800/1217380380 conn(0x7f868806dc40 legacy=0x7f8688070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 10 monclient: shutdown 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 >> v1:172.21.15.130:6789/0 conn(0x7f86a414be90 legacy=0x7f86a41db2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 shutdown_connections 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 >> 172.21.15.130:0/466280511 conn(0x7f86a40617a0 msgr2=0x7f86a413c660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:50.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 shutdown_connections 2024-03-25T19:47:50.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 -- 172.21.15.130:0/466280511 wait complete. 2024-03-25T19:47:50.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:50.729+0000 7f86ab9f6640 1 librados: shutdown 2024-03-25T19:47:50.741 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:47:57.548 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.662+0000 7f84e71c4640 10 monclient: get_monmap_and_config 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.662+0000 7f84e71c4640 10 monclient: build_initial_monmap 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.662+0000 7f84e71c4640 10 monclient: monmap: 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:47:57.663618+0000 2024-03-25T19:47:57.662 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:47:57.663618+0000 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.663+0000 7f84e71c4640 1 Processor -- start 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.663+0000 7f84e71c4640 1 -- start start 2024-03-25T19:47:57.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.663+0000 7f84e71c4640 10 monclient: init 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-a con 0x7f84e00a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-c con 0x7f84e013f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-b con 0x7f84e01430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e0142520 con 0x7f84e00a98f0 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00b28a0 con 0x7f84e013f710 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.664+0000 7f84e71c4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00b2a30 con 0x7f84e01430c0 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.665+0000 7f84e71c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:57.665 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.665+0000 7f84e71c4640 10 monclient(hunting): authenticate will time out at 5353.810651s 2024-03-25T19:47:57.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.677+0000 7f84e5761640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f84e01430c0 0x7f84e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:35716/0 (socket says 172.21.15.130:35716) 2024-03-25T19:47:57.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.677+0000 7f84e5761640 1 -- 172.21.15.130:0/3242384338 learned_addr learned my addr 172.21.15.130:0/3242384338 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267665390 0 0) 0x7f84e00b2a30 con 0x7f84e01430c0 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 10 monclient(hunting): my global_id is 4502 2024-03-25T19:47:57.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.687+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84d0003520 con 0x7f84e01430c0 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417191003 0 0) 0x7f84e00b28a0 con 0x7f84e013f710 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 10 monclient(hunting): my global_id is 4504 2024-03-25T19:47:57.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.688+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e00b2a30 con 0x7f84e013f710 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062119010 0 0) 0x7f84e0142520 con 0x7f84e00a98f0 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 10 monclient(hunting): my global_id is 4512 2024-03-25T19:47:57.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.695+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e00b28a0 con 0x7f84e00a98f0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649016558 0 0) 0x7f84d0003520 con 0x7f84e01430c0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 >> v1:172.21.15.163:6789/0 conn(0x7f84e013f710 legacy=0x7f84e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 >> v1:172.21.15.130:6789/0 conn(0x7f84e00a98f0 legacy=0x7f84e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 monclient: found mon.noname-b 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e00b44c0 con 0x7f84e01430c0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: _finish_auth 0 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:47:57.701 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f84d8001a10 con 0x7f84e01430c0 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84df7fe640 10 monclient: dump: 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84e71c4640 5 monclient: authenticate success, global_id 4502 2024-03-25T19:47:57.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.701+0000 7f84e71c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:47:57.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d8001b70 con 0x7f84e01430c0 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 1 -- 172.21.15.130:0/3242384338 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f84d8001f80 con 0x7f84e01430c0 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84df7fe640 10 monclient: dump: 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 10 monclient: get_monmap_and_config success 2024-03-25T19:47:57.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 4 set_mon_vals no callback set 2024-03-25T19:47:57.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 10 monclient: shutdown 2024-03-25T19:47:57.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 1 -- 172.21.15.130:0/3242384338 >> v1:172.21.15.130:6790/0 conn(0x7f84e01430c0 legacy=0x7f84e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 1 -- 172.21.15.130:0/3242384338 shutdown_connections 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 1 -- 172.21.15.130:0/3242384338 >> 172.21.15.130:0/3242384338 conn(0x7f84e00b7ea0 msgr2=0x7f84e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 1 -- 172.21.15.130:0/3242384338 shutdown_connections 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.720+0000 7f84e71c4640 1 -- 172.21.15.130:0/3242384338 wait complete. 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.721+0000 7f84e71c4640 10 monclient: build_initial_monmap 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.721+0000 7f84e71c4640 10 monclient: monmap: 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:47:57.723 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.721+0000 7f84e71c4640 1 librados: starting msgr at 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.721+0000 7f84e71c4640 1 librados: starting objecter 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 1 Processor -- start 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 1 -- start start 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 1 librados: setting wanted keys 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 1 librados: calling monclient init 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient: init 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-c con 0x7f84e00a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-a con 0x7f84e013f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.722+0000 7f84e71c4640 10 monclient(hunting): picked mon.noname-b con 0x7f84e01430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.723+0000 7f84e71c4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e01daca0 con 0x7f84e013f710 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.723+0000 7f84e71c4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e0138ab0 con 0x7f84e01430c0 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.723+0000 7f84e71c4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e00aba90 con 0x7f84e00a98f0 2024-03-25T19:47:57.724 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.723+0000 7f84e71c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:47:57.725 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.723+0000 7f84e71c4640 10 monclient(hunting): authenticate will time out at 5353.868729s 2024-03-25T19:47:57.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.731+0000 7f84dffff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f84e013f710 0x7f84e01db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:46914/0 (socket says 172.21.15.130:46914) 2024-03-25T19:47:57.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.731+0000 7f84dffff640 1 -- 172.21.15.130:0/130007983 learned_addr learned my addr 172.21.15.130:0/130007983 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:47:57.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.733+0000 7f84dd7fa640 10 client.?.objecter ms_handle_connect 0x7f84e013f710 2024-03-25T19:47:57.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.733+0000 7f84dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:57.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.734+0000 7f84dd7fa640 10 client.?.objecter ms_handle_connect 0x7f84e00a98f0 2024-03-25T19:47:57.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.734+0000 7f84dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:57.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.737+0000 7f84dd7fa640 10 client.?.objecter ms_handle_connect 0x7f84e01430c0 2024-03-25T19:47:57.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.737+0000 7f84dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:47:57.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.747+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 14410041 0 0) 0x7f84e01daca0 con 0x7f84e013f710 2024-03-25T19:47:57.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.747+0000 7f84dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.747+0000 7f84dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.747+0000 7f84dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.747+0000 7f84dd7fa640 10 monclient(hunting): my global_id is 4515 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84c4004eb0 con 0x7f84e013f710 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439341813 0 0) 0x7f84e0138ab0 con 0x7f84e01430c0 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.748 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 10 monclient(hunting): my global_id is 4507 2024-03-25T19:47:57.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.748+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e01daca0 con 0x7f84e01430c0 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876990782 0 0) 0x7f84e00aba90 con 0x7f84e00a98f0 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 10 monclient(hunting): my global_id is 4505 2024-03-25T19:47:57.751 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.751+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e0138ab0 con 0x7f84e00a98f0 2024-03-25T19:47:57.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 571529898 0 0) 0x7f84e0138ab0 con 0x7f84e00a98f0 2024-03-25T19:47:57.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e00aba90 con 0x7f84e00a98f0 2024-03-25T19:47:57.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f84d4001830 con 0x7f84e00a98f0 2024-03-25T19:47:57.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:57.767 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.767+0000 7f84dd7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.768+0000 7f84dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:47:57.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932757982 0 0) 0x7f84e01daca0 con 0x7f84e01430c0 2024-03-25T19:47:57.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f84e0138ab0 con 0x7f84e01430c0 2024-03-25T19:47:57.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f84d8001e80 con 0x7f84e01430c0 2024-03-25T19:47:57.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:57.769 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.770+0000 7f84dd7fa640 10 monclient(hunting): dump: 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:57.770 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3776158059 0 0) 0x7f84e00aba90 con 0x7f84e00a98f0 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 >> v1:172.21.15.163:6789/0 conn(0x7f84e01430c0 legacy=0x7f84e01d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 >> v1:172.21.15.130:6789/0 conn(0x7f84e013f710 legacy=0x7f84e01db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 1 monclient: found mon.c 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:47:57.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e01d41e0 con 0x7f84e00a98f0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 10 monclient: _finish_auth 0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 5 monclient: authenticate success, global_id 4505 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 10 monclient: _renew_subs 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e01db840 con 0x7f84e00a98f0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 10 monclient: _renew_subs 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e0138f00 con 0x7f84e00a98f0 2024-03-25T19:47:57.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.777+0000 7f84e71c4640 1 librados: init done 2024-03-25T19:47:57.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.778+0000 7f84e71c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:57.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.778+0000 7f84e71c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:57.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.778+0000 7f84e71c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:57.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:57.778+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e00627b0 con 0x7f84e00a98f0 2024-03-25T19:47:58.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f84d4001d40 con 0x7f84e00a98f0 2024-03-25T19:47:58.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:47:58.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f84d40037f0 con 0x7f84e00a98f0 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 10 monclient: dump: 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:47:58.094 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:47:58.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84deffd640 4 set_mon_vals no callback set 2024-03-25T19:47:58.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.094+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f84d4019880 con 0x7f84e00a98f0 2024-03-25T19:47:58.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.095+0000 7f84dffff640 10 monclient: get_auth_request con 0x7f84c406df80 auth_method 0 2024-03-25T19:47:58.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.095+0000 7f84dd7fa640 10 client.4505.objecter ms_handle_connect 0x7f84c406df80 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f84d4053580 con 0x7f84e00a98f0 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 10 client.4505.objecter ms_dispatch 0x7f84e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f84d404eec0 con 0x7f84e00a98f0 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:47:58.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.100+0000 7f84dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:47:58.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.351+0000 7f84e71c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.351+0000 7f84e71c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.351+0000 7f84e71c4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:47:58.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.351+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84e01d4370 con 0x7f84e00a98f0 2024-03-25T19:47:58.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.391+0000 7f84dd7fa640 1 -- 172.21.15.130:0/130007983 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f84d4050fc0 con 0x7f84e00a98f0 2024-03-25T19:47:58.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.391+0000 7f84dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:47:58.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.391+0000 7f84dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:47:58.391 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:47:58.391 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 10 librados: watch_flush enter 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 10 librados: watch_flush exit 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 10 client.4505.objecter successfully canceled tick 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 >> v1:172.21.15.163:6800/1217380380 conn(0x7f84c406df80 legacy=0x7f84c4070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 10 monclient: shutdown 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.393+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 >> v1:172.21.15.130:6790/0 conn(0x7f84e00a98f0 legacy=0x7f84e01383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.394+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 shutdown_connections 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.394+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 >> 172.21.15.130:0/130007983 conn(0x7f84e00b7ea0 msgr2=0x7f84e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:47:58.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.394+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 shutdown_connections 2024-03-25T19:47:58.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.394+0000 7f84e71c4640 1 -- 172.21.15.130:0/130007983 wait complete. 2024-03-25T19:47:58.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:47:58.394+0000 7f84e71c4640 1 librados: shutdown 2024-03-25T19:47:58.405 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:05.211 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:05.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.325+0000 7ff6f8250640 10 monclient: get_monmap_and_config 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.325+0000 7ff6f8250640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.325+0000 7ff6f8250640 10 monclient: monmap: 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:05.327779+0000 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:05.327779+0000 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.326+0000 7ff6f8250640 1 Processor -- start 2024-03-25T19:48:05.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.326+0000 7ff6f8250640 1 -- start start 2024-03-25T19:48:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.326+0000 7ff6f8250640 10 monclient: init 2024-03-25T19:48:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f00b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f00b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f014be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00af940 con 0x7ff6f00b4d60 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00b0a70 con 0x7ff6f00b1730 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f00b0d50 con 0x7ff6f014be90 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f8250640 10 monclient(hunting): authenticate will time out at 5361.473983s 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f67ed640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7ff6f014be90 0x7ff6f014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:40376/0 (socket says 172.21.15.130:40376) 2024-03-25T19:48:05.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.327+0000 7ff6f67ed640 1 -- 172.21.15.130:0/2098592046 learned_addr learned my addr 172.21.15.130:0/2098592046 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764068579 0 0) 0x7ff6f00b0d50 con 0x7ff6f014be90 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 10 monclient(hunting): my global_id is 4508 2024-03-25T19:48:05.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.349+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6dc003520 con 0x7ff6f014be90 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074898221 0 0) 0x7ff6f00b0a70 con 0x7ff6f00b1730 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 10 monclient(hunting): my global_id is 4510 2024-03-25T19:48:05.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.350+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f00b0d50 con 0x7ff6f00b1730 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2785429852 0 0) 0x7ff6f00af940 con 0x7ff6f00b4d60 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 10 monclient(hunting): my global_id is 4518 2024-03-25T19:48:05.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.356+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f00b0a70 con 0x7ff6f00b4d60 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 517927463 0 0) 0x7ff6dc003520 con 0x7ff6f014be90 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 >> v1:172.21.15.163:6789/0 conn(0x7ff6f00b1730 legacy=0x7ff6f00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 >> v1:172.21.15.130:6789/0 conn(0x7ff6f00b4d60 legacy=0x7ff6f013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 monclient: found mon.noname-b 2024-03-25T19:48:05.367 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f00b1030 con 0x7ff6f014be90 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff6e8001a10 con 0x7ff6f014be90 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f8250640 5 monclient: authenticate success, global_id 4508 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f8250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f4fea640 10 monclient: dump: 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.368 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.369 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.366+0000 7ff6f8250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:05.374 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e8001b70 con 0x7ff6f014be90 2024-03-25T19:48:05.374 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:05.374 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 1 -- 172.21.15.130:0/2098592046 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff6e8001f80 con 0x7ff6f014be90 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f4fea640 10 monclient: dump: 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f8250640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:05.375 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.374+0000 7ff6f8250640 4 set_mon_vals no callback set 2024-03-25T19:48:05.377 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 10 monclient: shutdown 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 1 -- 172.21.15.130:0/2098592046 >> v1:172.21.15.130:6790/0 conn(0x7ff6f014be90 legacy=0x7ff6f014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 1 -- 172.21.15.130:0/2098592046 shutdown_connections 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 1 -- 172.21.15.130:0/2098592046 >> 172.21.15.130:0/2098592046 conn(0x7ff6f00617a0 msgr2=0x7ff6f0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 1 -- 172.21.15.130:0/2098592046 shutdown_connections 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.375+0000 7ff6f8250640 1 -- 172.21.15.130:0/2098592046 wait complete. 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 10 monclient: build_initial_monmap 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 10 monclient: monmap: 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:05.378 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 librados: starting msgr at 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 librados: starting objecter 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 Processor -- start 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 -- start start 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 librados: setting wanted keys 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 1 librados: calling monclient init 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.376+0000 7ff6f8250640 10 monclient: init 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-a con 0x7ff6f00b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-c con 0x7ff6f00b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient(hunting): picked mon.noname-b con 0x7ff6f014be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:05.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0138490 con 0x7ff6f00b1730 2024-03-25T19:48:05.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0136cc0 con 0x7ff6f014be90 2024-03-25T19:48:05.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f0140520 con 0x7ff6f00b4d60 2024-03-25T19:48:05.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:05.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.377+0000 7ff6f8250640 10 monclient(hunting): authenticate will time out at 5361.523923s 2024-03-25T19:48:05.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.390+0000 7ff6f57eb640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7ff6f00b4d60 0x7ff6f0144e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:40402/0 (socket says 172.21.15.130:40402) 2024-03-25T19:48:05.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.390+0000 7ff6f57eb640 1 -- 172.21.15.130:0/1019839075 learned_addr learned my addr 172.21.15.130:0/1019839075 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:05.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.392+0000 7ff6e67fc640 10 client.?.objecter ms_handle_connect 0x7ff6f00b4d60 2024-03-25T19:48:05.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.392+0000 7ff6e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.392+0000 7ff6e67fc640 10 client.?.objecter ms_handle_connect 0x7ff6f00b1730 2024-03-25T19:48:05.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.392+0000 7ff6e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2893045397 0 0) 0x7ff6f0140520 con 0x7ff6f00b4d60 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 10 monclient(hunting): my global_id is 4511 2024-03-25T19:48:05.396 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.395+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d4004eb0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2434267510 0 0) 0x7ff6f0138490 con 0x7ff6f00b1730 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0140520 con 0x7ff6f00b1730 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 client.?.objecter ms_handle_connect 0x7ff6f014be90 2024-03-25T19:48:05.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.397+0000 7ff6e67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995838117 0 0) 0x7ff6f0136cc0 con 0x7ff6f014be90 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T19:48:05.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.406+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6f0138490 con 0x7ff6f014be90 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1201649012 0 0) 0x7ff6d4004eb0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6f0136cc0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff6e0001a30 con 0x7ff6f00b4d60 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 10 monclient(hunting): dump: 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.408 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.407+0000 7ff6e67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:05.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3563420948 0 0) 0x7ff6f0140520 con 0x7ff6f00b1730 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d4004eb0 con 0x7ff6f00b1730 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff6ec0018b0 con 0x7ff6f00b1730 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.409+0000 7ff6e67fc640 10 monclient(hunting): dump: 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:05.410 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:05.411 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:05.411 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2222324848 0 0) 0x7ff6f0136cc0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 >> v1:172.21.15.163:6789/0 conn(0x7ff6f014be90 legacy=0x7ff6f01e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 >> v1:172.21.15.130:6789/0 conn(0x7ff6f00b1730 legacy=0x7ff6f0140940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 1 monclient: found mon.c 2024-03-25T19:48:05.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f0138700 con 0x7ff6f00b4d60 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 5 monclient: authenticate success, global_id 4511 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 10 monclient: _renew_subs 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f0136660 con 0x7ff6f00b4d60 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 10 monclient: _renew_subs 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f0137530 con 0x7ff6f00b4d60 2024-03-25T19:48:05.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.423+0000 7ff6f8250640 1 librados: init done 2024-03-25T19:48:05.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.424+0000 7ff6f8250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.424+0000 7ff6f8250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.424+0000 7ff6f8250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.424+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f01369b0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6e0001c70 con 0x7ff6f00b4d60 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ff6e0003a60 con 0x7ff6f00b4d60 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e67fc640 10 monclient: dump: 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:05.733 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.732+0000 7ff6e7fff640 4 set_mon_vals no callback set 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.733+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ff6e0003c40 con 0x7ff6f00b4d60 2024-03-25T19:48:05.734 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.734+0000 7ff6f5fec640 10 monclient: get_auth_request con 0x7ff6d406df70 auth_method 0 2024-03-25T19:48:05.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.734+0000 7ff6e67fc640 10 client.4511.objecter ms_handle_connect 0x7ff6d406df70 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ff6e0054890 con 0x7ff6f00b4d60 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 10 client.4511.objecter ms_dispatch 0x7ff6f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6e00503a0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:05.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.738+0000 7ff6e67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:05.781 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:50.714 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.837 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:50.770 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.837 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:50.770 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.844 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:50.777 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.865 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:06:50.798 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.879 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:06:50.812 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.908 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:50.809 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.914 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:50.815 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.947 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:06:50.848 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:05.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.952+0000 7ff6f8250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:05.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.952+0000 7ff6f8250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:05.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.952+0000 7ff6f8250640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:05.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.952+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6f00627b0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.993+0000 7ff6e67fc640 1 -- 172.21.15.130:0/1019839075 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ff6e00522d0 con 0x7ff6f00b4d60 2024-03-25T19:48:05.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.993+0000 7ff6e67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:05.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.993+0000 7ff6e67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:05.994 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:05.994 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 10 librados: watch_flush enter 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 10 librados: watch_flush exit 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 10 client.4511.objecter successfully canceled tick 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 >> v1:172.21.15.163:6800/1217380380 conn(0x7ff6d406df70 legacy=0x7ff6d4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 10 monclient: shutdown 2024-03-25T19:48:05.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 >> v1:172.21.15.130:6790/0 conn(0x7ff6f00b4d60 legacy=0x7ff6f0144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 shutdown_connections 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 >> 172.21.15.130:0/1019839075 conn(0x7ff6f00617a0 msgr2=0x7ff6f00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 shutdown_connections 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 -- 172.21.15.130:0/1019839075 wait complete. 2024-03-25T19:48:05.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:05.995+0000 7ff6f8250640 1 librados: shutdown 2024-03-25T19:48:06.006 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:12.814 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.926+0000 7ffaf3c4e640 10 monclient: get_monmap_and_config 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.926+0000 7ffaf3c4e640 10 monclient: build_initial_monmap 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.926+0000 7ffaf3c4e640 10 monclient: monmap: 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:12.927506+0000 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:12.927506+0000 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:12.926 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:12.927 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:12.927 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:12.927 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:12.927 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.927+0000 7ffaf3c4e640 1 Processor -- start 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.927+0000 7ffaf3c4e640 1 -- start start 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.927+0000 7ffaf3c4e640 10 monclient: init 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec0b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec0b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0af940 con 0x7ffaec0b1730 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b0a70 con 0x7ffaec0b4d60 2024-03-25T19:48:12.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec0b0d50 con 0x7ffaec14be90 2024-03-25T19:48:12.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:12.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.928+0000 7ffaf3c4e640 10 monclient(hunting): authenticate will time out at 5369.073863s 2024-03-25T19:48:12.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.940+0000 7ffaf19ea640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffaec0b4d60 0x7ffaec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:60246/0 (socket says 172.21.15.130:60246) 2024-03-25T19:48:12.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.940+0000 7ffaf19ea640 1 -- 172.21.15.130:0/1614284456 learned_addr learned my addr 172.21.15.130:0/1614284456 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.948+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704501313 0 0) 0x7ffaec0b0a70 con 0x7ffaec0b4d60 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.949+0000 7ffaf09e8640 10 monclient(hunting): _negotiate 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.949+0000 7ffaf09e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.949+0000 7ffaf09e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.949+0000 7ffaf09e8640 10 monclient(hunting): my global_id is 4516 2024-03-25T19:48:12.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.949+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad8003520 con 0x7ffaec0b4d60 2024-03-25T19:48:12.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1934128815 0 0) 0x7ffad8003520 con 0x7ffaec0b4d60 2024-03-25T19:48:12.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 >> v1:172.21.15.130:6790/0 conn(0x7ffaec14be90 legacy=0x7ffaec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:12.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 >> v1:172.21.15.130:6789/0 conn(0x7ffaec0b1730 legacy=0x7ffaec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:12.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: _finish_hunting 0 2024-03-25T19:48:12.963 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 monclient: found mon.noname-c 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec0b1030 con 0x7ffaec0b4d60 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: _finish_auth 0 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: _check_auth_tickets 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffae8001690 con 0x7ffaec0b4d60 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf09e8640 10 monclient: dump: 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:12.964 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf3c4e640 5 monclient: authenticate success, global_id 4516 2024-03-25T19:48:12.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.963+0000 7ffaf3c4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae80017f0 con 0x7ffaec0b4d60 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 1 -- 172.21.15.130:0/1614284456 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffae8001c00 con 0x7ffaec0b4d60 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf3c4e640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf3c4e640 4 set_mon_vals no callback set 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:12.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.982+0000 7ffaf09e8640 10 monclient: dump: 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:12.983 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 10 monclient: shutdown 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/1614284456 >> v1:172.21.15.163:6789/0 conn(0x7ffaec0b4d60 legacy=0x7ffaec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/1614284456 shutdown_connections 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/1614284456 >> 172.21.15.130:0/1614284456 conn(0x7ffaec0617a0 msgr2=0x7ffaec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:12.985 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/1614284456 shutdown_connections 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.983+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/1614284456 wait complete. 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 10 monclient: build_initial_monmap 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 10 monclient: monmap: 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:12.986 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 librados: starting msgr at 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 librados: starting objecter 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 Processor -- start 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 -- start start 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 librados: setting wanted keys 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 1 librados: calling monclient init 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.984+0000 7ffaf3c4e640 10 monclient: init 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-b con 0x7ffaec0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-a con 0x7ffaec0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient(hunting): picked mon.noname-c con 0x7ffaec14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:12.987 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec144150 con 0x7ffaec0b4d60 2024-03-25T19:48:12.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec145840 con 0x7ffaec0b1730 2024-03-25T19:48:12.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffaec143c30 con 0x7ffaec14be90 2024-03-25T19:48:12.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:12.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:12.985+0000 7ffaf3c4e640 10 monclient(hunting): authenticate will time out at 5369.131303s 2024-03-25T19:48:13.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.001+0000 7ffaf11e9640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ffaec0b4d60 0x7ffaec1d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:53126/0 (socket says 172.21.15.130:53126) 2024-03-25T19:48:13.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.001+0000 7ffaf11e9640 1 -- 172.21.15.130:0/4201942625 learned_addr learned my addr 172.21.15.130:0/4201942625 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:13.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.002+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec0b4d60 2024-03-25T19:48:13.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.002+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.006+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec0b1730 2024-03-25T19:48:13.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.006+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232702982 0 0) 0x7ffaec145840 con 0x7ffaec0b1730 2024-03-25T19:48:13.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T19:48:13.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.009+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec0b1730 2024-03-25T19:48:13.016 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183536596 0 0) 0x7ffaec144150 con 0x7ffaec0b4d60 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4527 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.017+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffaec145840 con 0x7ffaec0b4d60 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.018+0000 7ffae27fc640 10 client.?.objecter ms_handle_connect 0x7ffaec14be90 2024-03-25T19:48:13.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.018+0000 7ffae27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809556731 0 0) 0x7ffad0004eb0 con 0x7ffaec0b1730 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaec144150 con 0x7ffaec0b1730 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffae8001b00 con 0x7ffaec0b1730 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:13.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:13.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.025+0000 7ffae27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:13.029 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807677910 0 0) 0x7ffaec143c30 con 0x7ffaec14be90 2024-03-25T19:48:13.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:13.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:13.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:13.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T19:48:13.030 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.030+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad0004eb0 con 0x7ffaec14be90 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3340094795 0 0) 0x7ffaec145840 con 0x7ffaec0b4d60 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffaec143c30 con 0x7ffaec0b4d60 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffadc0019f0 con 0x7ffaec0b4d60 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.037+0000 7ffae27fc640 10 monclient(hunting): dump: 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:13.037 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:13.038 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4053987782 0 0) 0x7ffad0004eb0 con 0x7ffaec14be90 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad0005ed0 con 0x7ffaec14be90 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 815733459 0 0) 0x7ffaec144150 con 0x7ffaec0b1730 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 >> v1:172.21.15.130:6790/0 conn(0x7ffaec14be90 legacy=0x7ffaec1e2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 >> v1:172.21.15.130:6789/0 conn(0x7ffaec0b4d60 legacy=0x7ffaec1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 monclient: found mon.b 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaec143900 con 0x7ffaec0b1730 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:13.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffaf3c4e640 5 monclient: authenticate success, global_id 4519 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffaf3c4e640 10 monclient: _renew_subs 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffaf3c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.063+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaec137990 con 0x7ffaec0b1730 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 monclient: _renew_subs 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaec1307e0 con 0x7ffaec0b1730 2024-03-25T19:48:13.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 1 librados: init done 2024-03-25T19:48:13.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:13.066 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.064+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaec1401f0 con 0x7ffaec0b1730 2024-03-25T19:48:13.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffae8004a90 con 0x7ffaec0b1730 2024-03-25T19:48:13.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:13.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffae8004da0 con 0x7ffaec0b1730 2024-03-25T19:48:13.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:13.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae27fc640 10 monclient: dump: 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:13.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.355+0000 7ffae3fff640 4 set_mon_vals no callback set 2024-03-25T19:48:13.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.356+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ffae801ad70 con 0x7ffaec0b1730 2024-03-25T19:48:13.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.357+0000 7ffaf11e9640 10 monclient: get_auth_request con 0x7ffad006e430 auth_method 0 2024-03-25T19:48:13.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.357+0000 7ffae27fc640 10 client.4519.objecter ms_handle_connect 0x7ffad006e430 2024-03-25T19:48:13.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.362+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7ffae80547f0 con 0x7ffaec0b1730 2024-03-25T19:48:13.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.362+0000 7ffae27fc640 10 client.4519.objecter ms_dispatch 0x7ffaec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.362+0000 7ffae27fc640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.362+0000 7ffae27fc640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.362+0000 7ffae27fc640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.363+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffae8050130 con 0x7ffaec0b1730 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.363+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:13.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.363+0000 7ffae27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:13.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.583+0000 7ffaf3c4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:13.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.583+0000 7ffaf3c4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:13.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.583+0000 7ffaf3c4e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:13.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.583+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffaec143e20 con 0x7ffaec0b1730 2024-03-25T19:48:13.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.611+0000 7ffae27fc640 1 -- 172.21.15.130:0/4201942625 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7ffae8052230 con 0x7ffaec0b1730 2024-03-25T19:48:13.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.611+0000 7ffae27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:13.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.611+0000 7ffae27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:13.611 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:13.611 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:13.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 10 librados: watch_flush enter 2024-03-25T19:48:13.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 10 librados: watch_flush exit 2024-03-25T19:48:13.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 10 client.4519.objecter successfully canceled tick 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 >> v1:172.21.15.163:6800/1217380380 conn(0x7ffad006e430 legacy=0x7ffad00708f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 10 monclient: shutdown 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 >> v1:172.21.15.163:6789/0 conn(0x7ffaec0b1730 legacy=0x7ffaec12fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 shutdown_connections 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 >> 172.21.15.130:0/4201942625 conn(0x7ffaec0617a0 msgr2=0x7ffaec13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 shutdown_connections 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 -- 172.21.15.130:0/4201942625 wait complete. 2024-03-25T19:48:13.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:13.614+0000 7ffaf3c4e640 1 librados: shutdown 2024-03-25T19:48:13.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:20.433 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:20.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.550+0000 7f7349b7b640 10 monclient: get_monmap_and_config 2024-03-25T19:48:20.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.550+0000 7f7349b7b640 10 monclient: build_initial_monmap 2024-03-25T19:48:20.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.550+0000 7f7349b7b640 10 monclient: monmap: 2024-03-25T19:48:20.550 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:20.550 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:20.551577+0000 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:20.551577+0000 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:20.551 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.552 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.551+0000 7f7349b7b640 1 Processor -- start 2024-03-25T19:48:20.552 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.551+0000 7f7349b7b640 1 -- start start 2024-03-25T19:48:20.552 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.551+0000 7f7349b7b640 10 monclient: init 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f73440a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f734413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f73441430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7344142520 con 0x7f734413f710 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440b28a0 con 0x7f73440a98f0 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73440b2a30 con 0x7f73441430c0 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:20.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.552+0000 7f7349b7b640 10 monclient(hunting): authenticate will time out at 5376.698313s 2024-03-25T19:48:20.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.555+0000 7f73437fe640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f73440a98f0 0x7f73440a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:50732/0 (socket says 172.21.15.130:50732) 2024-03-25T19:48:20.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.555+0000 7f73437fe640 1 -- 172.21.15.130:0/529990340 learned_addr learned my addr 172.21.15.130:0/529990340 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:20.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218441182 0 0) 0x7f73440b2a30 con 0x7f73441430c0 2024-03-25T19:48:20.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 10 monclient(hunting): my global_id is 4520 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.573+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f732c003520 con 0x7f73441430c0 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4079388796 0 0) 0x7f73440b28a0 con 0x7f73440a98f0 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T19:48:20.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.574+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73440b2a30 con 0x7f73440a98f0 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.584+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 761789553 0 0) 0x7f732c003520 con 0x7f73441430c0 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 >> v1:172.21.15.163:6789/0 conn(0x7f73440a98f0 legacy=0x7f73440a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 >> v1:172.21.15.130:6789/0 conn(0x7f734413f710 legacy=0x7f73441429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 1 monclient: found mon.noname-b 2024-03-25T19:48:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73440b44c0 con 0x7f73441430c0 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f73427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f7349b7b640 5 monclient: authenticate success, global_id 4520 2024-03-25T19:48:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.585+0000 7f7349b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.586+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f733c0019d0 con 0x7f73441430c0 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.586+0000 7f73427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.586+0000 7f73427fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.586+0000 7f73427fc640 10 monclient: dump: 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:20.586 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.586+0000 7f7349b7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:20.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f733c001c20 con 0x7f73441430c0 2024-03-25T19:48:20.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:20.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 1 -- 172.21.15.130:0/529990340 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f733c001fb0 con 0x7f73441430c0 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 4 set_mon_vals no callback set 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f73427fc640 10 monclient: dump: 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:20.595 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 10 monclient: shutdown 2024-03-25T19:48:20.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:20.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 1 -- 172.21.15.130:0/529990340 >> v1:172.21.15.130:6790/0 conn(0x7f73441430c0 legacy=0x7f734414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 1 -- 172.21.15.130:0/529990340 shutdown_connections 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.595+0000 7f7349b7b640 1 -- 172.21.15.130:0/529990340 >> 172.21.15.130:0/529990340 conn(0x7f73440b7ea0 msgr2=0x7f73440b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.596+0000 7f7349b7b640 1 -- 172.21.15.130:0/529990340 shutdown_connections 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.596+0000 7f7349b7b640 1 -- 172.21.15.130:0/529990340 wait complete. 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.596+0000 7f7349b7b640 10 monclient: build_initial_monmap 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.596+0000 7f7349b7b640 10 monclient: monmap: 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:20.597 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:20.598 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 librados: starting msgr at 2024-03-25T19:48:20.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 librados: starting objecter 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 Processor -- start 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 -- start start 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 librados: setting wanted keys 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 1 librados: calling monclient init 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.597+0000 7f7349b7b640 10 monclient: init 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-b con 0x7f73440a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-a con 0x7f734413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient(hunting): picked mon.noname-c con 0x7f73441430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:20.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734406fcd0 con 0x7f734413f710 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f734406ff40 con 0x7f73440a98f0 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7344070220 con 0x7f73441430c0 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.598+0000 7f7349b7b640 10 monclient(hunting): authenticate will time out at 5376.744154s 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.600+0000 7f7342ffd640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f734413f710 0x7f734406eea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:41386/0 (socket says 172.21.15.130:41386) 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.600+0000 7f7342ffd640 1 -- 172.21.15.130:0/128638545 learned_addr learned my addr 172.21.15.130:0/128638545 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.601+0000 7f732bfff640 10 client.?.objecter ms_handle_connect 0x7f734413f710 2024-03-25T19:48:20.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.601+0000 7f732bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:20.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173609749 0 0) 0x7f734406fcd0 con 0x7f734413f710 2024-03-25T19:48:20.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 10 monclient(hunting): my global_id is 4533 2024-03-25T19:48:20.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.606+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7330004eb0 con 0x7f734413f710 2024-03-25T19:48:20.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.609+0000 7f732bfff640 10 client.?.objecter ms_handle_connect 0x7f73440a98f0 2024-03-25T19:48:20.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.609+0000 7f732bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:20.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.609+0000 7f732bfff640 10 client.?.objecter ms_handle_connect 0x7f73441430c0 2024-03-25T19:48:20.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.609+0000 7f732bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158029938 0 0) 0x7f7344070220 con 0x7f73441430c0 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 10 monclient(hunting): my global_id is 4523 2024-03-25T19:48:20.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.615+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f734406fcd0 con 0x7f73441430c0 2024-03-25T19:48:20.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652121700 0 0) 0x7f734406ff40 con 0x7f73440a98f0 2024-03-25T19:48:20.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:48:20.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:20.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 10 monclient(hunting): my global_id is 4525 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.626+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7344070220 con 0x7f73440a98f0 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2301055353 0 0) 0x7f7330004eb0 con 0x7f734413f710 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f734406ff40 con 0x7f734413f710 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f7338001a30 con 0x7f734413f710 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 10 monclient(hunting): dump: 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.627 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.628 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.627+0000 7f732bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:20.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732534809 0 0) 0x7f734406fcd0 con 0x7f73441430c0 2024-03-25T19:48:20.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7330004eb0 con 0x7f73441430c0 2024-03-25T19:48:20.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f733c001f30 con 0x7f73441430c0 2024-03-25T19:48:20.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:20.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.629+0000 7f732bfff640 10 monclient(hunting): dump: 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:20.630 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.636 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1016572142 0 0) 0x7f7330004eb0 con 0x7f73441430c0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 >> v1:172.21.15.163:6789/0 conn(0x7f73440a98f0 legacy=0x7f734414bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 >> v1:172.21.15.130:6789/0 conn(0x7f734413f710 legacy=0x7f734406eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 1 monclient: found mon.c 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.636+0000 7f732bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f732bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7344070500 con 0x7f73441430c0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f732bfff640 10 monclient: _finish_auth 0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f732bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f732bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 5 monclient: authenticate success, global_id 4523 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 10 monclient: _renew_subs 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:20.637 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7344071e10 con 0x7f73441430c0 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 10 monclient: _renew_subs 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7344071bd0 con 0x7f73441430c0 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.637+0000 7f7349b7b640 1 librados: init done 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.638+0000 7f7349b7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:20.638 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.638+0000 7f7349b7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:20.639 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.638+0000 7f7349b7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:20.639 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.638+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7344068290 con 0x7f73441430c0 2024-03-25T19:48:20.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.923+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f733c002090 con 0x7f73441430c0 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.923+0000 7f732bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.923+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f733c003c20 con 0x7f73441430c0 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.924+0000 7f732bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.924+0000 7f732bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.924+0000 7f732bfff640 10 monclient: dump: 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:20.924 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.924+0000 7f7341ffb640 4 set_mon_vals no callback set 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.925+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f733c019bf0 con 0x7f73441430c0 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.925+0000 7f73437fe640 10 monclient: get_auth_request con 0x7f733006df60 auth_method 0 2024-03-25T19:48:20.925 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.925+0000 7f732bfff640 10 client.4523.objecter ms_handle_connect 0x7f733006df60 2024-03-25T19:48:20.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.929+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f733c053790 con 0x7f73441430c0 2024-03-25T19:48:20.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.929+0000 7f732bfff640 10 client.4523.objecter ms_dispatch 0x7f7344000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:20.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.929+0000 7f732bfff640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:20.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.929+0000 7f732bfff640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:20.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.929+0000 7f732bfff640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T19:48:20.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.930+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f733c04f0d0 con 0x7f73441430c0 2024-03-25T19:48:20.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.930+0000 7f732bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:20.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:20.930+0000 7f732bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:21.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.176+0000 7f7349b7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.176+0000 7f7349b7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.176+0000 7f7349b7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:21.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.176+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73440627b0 con 0x7f73441430c0 2024-03-25T19:48:21.250 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.250+0000 7f732bfff640 1 -- 172.21.15.130:0/128638545 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f733c0511d0 con 0x7f73441430c0 2024-03-25T19:48:21.250 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.250+0000 7f732bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:21.250 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.250+0000 7f732bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:21.250 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:21.251 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:21.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 10 librados: watch_flush enter 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 10 librados: watch_flush exit 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 10 client.4523.objecter successfully canceled tick 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 >> v1:172.21.15.163:6800/1217380380 conn(0x7f733006df60 legacy=0x7f7330070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 10 monclient: shutdown 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 >> v1:172.21.15.130:6790/0 conn(0x7f73441430c0 legacy=0x7f73441e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.253+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 shutdown_connections 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.254+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 >> 172.21.15.130:0/128638545 conn(0x7f73440b7ea0 msgr2=0x7f7344074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.254+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 shutdown_connections 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.254+0000 7f7349b7b640 1 -- 172.21.15.130:0/128638545 wait complete. 2024-03-25T19:48:21.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:21.254+0000 7f7349b7b640 1 librados: shutdown 2024-03-25T19:48:21.266 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:25.596 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:10.529 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.653 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:10.586 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.693 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:10.626 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.703 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:10.636 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.708 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:10.609 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.710 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:10.643 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.717 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:10.618 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.740 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:10.673 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:25.747 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:10.648 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:28.073 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:28.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.187+0000 7f03dfc57640 10 monclient: get_monmap_and_config 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.187+0000 7f03dfc57640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.187+0000 7f03dfc57640 10 monclient: monmap: 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:28.188019+0000 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:28.188019+0000 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:28.187 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.187+0000 7f03dfc57640 1 Processor -- start 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 1 -- start start 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient: init 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-c con 0x7f03d80b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-b con 0x7f03d80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-a con 0x7f03d814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d80af940 con 0x7f03d814be90 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d80b0a70 con 0x7f03d80b4d60 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d80b0d50 con 0x7f03d80b1730 2024-03-25T19:48:28.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.188+0000 7f03dfc57640 10 monclient(hunting): authenticate will time out at 5384.334499s 2024-03-25T19:48:28.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.190+0000 7f03dd1f2640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f03d80b1730 0x7f03d80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:47454/0 (socket says 172.21.15.130:47454) 2024-03-25T19:48:28.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.190+0000 7f03dd1f2640 1 -- 172.21.15.130:0/1078873851 learned_addr learned my addr 172.21.15.130:0/1078873851 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620391250 0 0) 0x7f03d80b0d50 con 0x7f03d80b1730 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 10 monclient(hunting): my global_id is 4526 2024-03-25T19:48:28.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.195+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03c4003520 con 0x7f03d80b1730 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550650482 0 0) 0x7f03c4003520 con 0x7f03d80b1730 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 >> v1:172.21.15.163:6789/0 conn(0x7f03d80b4d60 legacy=0x7f03d813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 >> v1:172.21.15.130:6789/0 conn(0x7f03d814be90 legacy=0x7f03d814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: _finish_hunting 0 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 monclient: found mon.noname-b 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d80b1030 con 0x7f03d80b1730 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: _finish_auth 0 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: _check_auth_tickets 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:28.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03c8001690 con 0x7f03d80b1730 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dc9f1640 10 monclient: dump: 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.203 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.204 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dfc57640 5 monclient: authenticate success, global_id 4526 2024-03-25T19:48:28.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.202+0000 7f03dfc57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03c8001860 con 0x7f03d80b1730 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 1 -- 172.21.15.130:0/1078873851 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03c8001cb0 con 0x7f03d80b1730 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dfc57640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dfc57640 4 set_mon_vals no callback set 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.211+0000 7f03dc9f1640 10 monclient: dump: 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.212 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 10 monclient: shutdown 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 1 -- 172.21.15.130:0/1078873851 >> v1:172.21.15.130:6790/0 conn(0x7f03d80b1730 legacy=0x7f03d80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 1 -- 172.21.15.130:0/1078873851 shutdown_connections 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 1 -- 172.21.15.130:0/1078873851 >> 172.21.15.130:0/1078873851 conn(0x7f03d80617a0 msgr2=0x7f03d8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 1 -- 172.21.15.130:0/1078873851 shutdown_connections 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.212+0000 7f03dfc57640 1 -- 172.21.15.130:0/1078873851 wait complete. 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.213+0000 7f03dfc57640 10 monclient: build_initial_monmap 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.213+0000 7f03dfc57640 10 monclient: monmap: 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:28.214 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 librados: starting msgr at 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 librados: starting objecter 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 Processor -- start 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 -- start start 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 librados: setting wanted keys 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 1 librados: calling monclient init 2024-03-25T19:48:28.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.214+0000 7f03dfc57640 10 monclient: init 2024-03-25T19:48:28.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-c con 0x7f03d80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-b con 0x7f03d80b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient(hunting): picked mon.noname-a con 0x7f03d814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d813b320 con 0x7f03d814be90 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d813b600 con 0x7f03d80b4d60 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d813b8e0 con 0x7f03d80b1730 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:28.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.215+0000 7f03dfc57640 10 monclient(hunting): authenticate will time out at 5384.361488s 2024-03-25T19:48:28.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.226+0000 7f03dd9f3640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f03d80b1730 0x7f03d813a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:47472/0 (socket says 172.21.15.130:47472) 2024-03-25T19:48:28.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.226+0000 7f03dd9f3640 1 -- 172.21.15.130:0/2250485144 learned_addr learned my addr 172.21.15.130:0/2250485144 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:28.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.233+0000 7f03ce7fc640 10 client.?.objecter ms_handle_connect 0x7f03d80b4d60 2024-03-25T19:48:28.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.233+0000 7f03ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.237+0000 7f03ce7fc640 10 client.?.objecter ms_handle_connect 0x7f03d80b1730 2024-03-25T19:48:28.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.237+0000 7f03ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.237+0000 7f03ce7fc640 10 client.?.objecter ms_handle_connect 0x7f03d814be90 2024-03-25T19:48:28.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.237+0000 7f03ce7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:28.245 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424469275 0 0) 0x7f03d813b600 con 0x7f03d80b4d60 2024-03-25T19:48:28.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 10 monclient(hunting): my global_id is 4528 2024-03-25T19:48:28.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.246+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03bc004eb0 con 0x7f03d80b4d60 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531059839 0 0) 0x7f03d813b8e0 con 0x7f03d80b1730 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 10 monclient(hunting): my global_id is 4529 2024-03-25T19:48:28.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.248+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d813b600 con 0x7f03d80b1730 2024-03-25T19:48:28.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4028139839 0 0) 0x7f03d813b320 con 0x7f03d814be90 2024-03-25T19:48:28.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:48:28.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:28.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:28.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T19:48:28.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.256+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03d813b8e0 con 0x7f03d814be90 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3001680207 0 0) 0x7f03d813b8e0 con 0x7f03d814be90 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03d813b320 con 0x7f03d814be90 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03d0001a10 con 0x7f03d814be90 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.262 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.263 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.262+0000 7f03ce7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.263+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3477029591 0 0) 0x7f03bc004eb0 con 0x7f03d80b4d60 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.264+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03d813b8e0 con 0x7f03d80b4d60 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.264+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03c8001b70 con 0x7f03d80b4d60 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.264+0000 7f03ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.264+0000 7f03ce7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.264+0000 7f03ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.264 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.265 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.265 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.265 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 441963160 0 0) 0x7f03d813b600 con 0x7f03d80b1730 2024-03-25T19:48:28.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f03bc004eb0 con 0x7f03d80b1730 2024-03-25T19:48:28.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03d4004000 con 0x7f03d80b1730 2024-03-25T19:48:28.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.268 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.269+0000 7f03ce7fc640 10 monclient(hunting): dump: 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.269 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 109737937 0 0) 0x7f03d813b320 con 0x7f03d814be90 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 >> v1:172.21.15.130:6790/0 conn(0x7f03d80b1730 legacy=0x7f03d813a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 >> v1:172.21.15.163:6789/0 conn(0x7f03d80b4d60 legacy=0x7f03d8140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 1 monclient: found mon.a 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d8133790 con 0x7f03d814be90 2024-03-25T19:48:28.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 10 monclient: _finish_auth 0 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 5 monclient: authenticate success, global_id 4536 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 10 monclient: _renew_subs 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d813bd00 con 0x7f03d814be90 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 10 monclient: _renew_subs 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d8140c70 con 0x7f03d814be90 2024-03-25T19:48:28.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.271+0000 7f03dfc57640 1 librados: init done 2024-03-25T19:48:28.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.272+0000 7f03dfc57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:28.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.272+0000 7f03dfc57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:28.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.272+0000 7f03dfc57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:48:28.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.272+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d80627b0 con 0x7f03d814be90 2024-03-25T19:48:28.581 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f03d0001f20 con 0x7f03d814be90 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f03d0003b10 con 0x7f03d814be90 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 10 monclient: dump: 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:28.582 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03cffff640 4 set_mon_vals no callback set 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.582+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f03d0003d30 con 0x7f03d814be90 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.583+0000 7f03dd9f3640 10 monclient: get_auth_request con 0x7f03bc06e0b0 auth_method 0 2024-03-25T19:48:28.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.583+0000 7f03ce7fc640 10 client.4536.objecter ms_handle_connect 0x7f03bc06e0b0 2024-03-25T19:48:28.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f03d0053650 con 0x7f03d814be90 2024-03-25T19:48:28.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 10 client.4536.objecter ms_dispatch 0x7f03d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:28.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:28.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:28.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-25T19:48:28.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f03d004f160 con 0x7f03d814be90 2024-03-25T19:48:28.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:28.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.589+0000 7f03ce7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:28.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.841+0000 7f03dfc57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:28.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.841+0000 7f03dfc57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:28.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.841+0000 7f03dfc57640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:48:28.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.841+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d813e7d0 con 0x7f03d814be90 2024-03-25T19:48:28.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.863+0000 7f03ce7fc640 1 -- 172.21.15.130:0/2250485144 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f03d0051090 con 0x7f03d814be90 2024-03-25T19:48:28.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.863+0000 7f03ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.863+0000 7f03ce7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:28.863 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:28.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 10 librados: watch_flush enter 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 10 librados: watch_flush exit 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 10 client.4536.objecter successfully canceled tick 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 >> v1:172.21.15.163:6800/1217380380 conn(0x7f03bc06e0b0 legacy=0x7f03bc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 10 monclient: shutdown 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 >> v1:172.21.15.130:6789/0 conn(0x7f03d814be90 legacy=0x7f03d81e64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.865+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 shutdown_connections 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.866+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 >> 172.21.15.130:0/2250485144 conn(0x7f03d80617a0 msgr2=0x7f03d80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.866+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 shutdown_connections 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.866+0000 7f03dfc57640 1 -- 172.21.15.130:0/2250485144 wait complete. 2024-03-25T19:48:28.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:28.866+0000 7f03dfc57640 1 librados: shutdown 2024-03-25T19:48:28.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:35.684 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.803+0000 7f00243a3640 10 monclient: get_monmap_and_config 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.803+0000 7f00243a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.803+0000 7f00243a3640 10 monclient: monmap: 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:35.804448+0000 2024-03-25T19:48:35.803 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:35.804448+0000 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:35.804 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.804+0000 7f00243a3640 1 Processor -- start 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.804+0000 7f00243a3640 1 -- start start 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.804+0000 7f00243a3640 10 monclient: init 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:35.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-a con 0x7f001c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-c con 0x7f001c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-b con 0x7f001c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c142520 con 0x7f001c0a98f0 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0b28a0 con 0x7f001c13f710 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c0b2a30 con 0x7f001c1430c0 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:35.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.805+0000 7f00243a3640 10 monclient(hunting): authenticate will time out at 5391.950948s 2024-03-25T19:48:35.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.812+0000 7f002193e640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f001c13f710 0x7f001c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:42152/0 (socket says 172.21.15.130:42152) 2024-03-25T19:48:35.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.812+0000 7f002193e640 1 -- 172.21.15.130:0/890303330 learned_addr learned my addr 172.21.15.130:0/890303330 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:35.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2538062858 0 0) 0x7f001c0b28a0 con 0x7f001c13f710 2024-03-25T19:48:35.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 10 monclient(hunting): my global_id is 4531 2024-03-25T19:48:35.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.829+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008003520 con 0x7f001c13f710 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645221311 0 0) 0x7f001c142520 con 0x7f001c0a98f0 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 10 monclient(hunting): my global_id is 4539 2024-03-25T19:48:35.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.832+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c0b28a0 con 0x7f001c0a98f0 2024-03-25T19:48:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146948828 0 0) 0x7f001c0b2a30 con 0x7f001c1430c0 2024-03-25T19:48:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 10 monclient(hunting): my global_id is 4532 2024-03-25T19:48:35.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.843+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c142520 con 0x7f001c1430c0 2024-03-25T19:48:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 747497408 0 0) 0x7f0008003520 con 0x7f001c13f710 2024-03-25T19:48:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 >> v1:172.21.15.130:6790/0 conn(0x7f001c1430c0 legacy=0x7f001c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 >> v1:172.21.15.130:6789/0 conn(0x7f001c0a98f0 legacy=0x7f001c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:35.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: _finish_hunting 0 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 monclient: found mon.noname-c 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c0b44c0 con 0x7f001c13f710 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: _finish_auth 0 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: _check_auth_tickets 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0018001690 con 0x7f001c13f710 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f00243a3640 5 monclient: authenticate success, global_id 4531 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f00243a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f002113d640 10 monclient: dump: 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.847 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.848 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.846+0000 7f00243a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:35.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00180017f0 con 0x7f001c13f710 2024-03-25T19:48:35.856 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 1 -- 172.21.15.130:0/890303330 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0018001c00 con 0x7f001c13f710 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f002113d640 10 monclient: dump: 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f00243a3640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:35.857 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f00243a3640 4 set_mon_vals no callback set 2024-03-25T19:48:35.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f00243a3640 10 monclient: shutdown 2024-03-25T19:48:35.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f00243a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.857+0000 7f00243a3640 1 -- 172.21.15.130:0/890303330 >> v1:172.21.15.163:6789/0 conn(0x7f001c13f710 legacy=0x7f001c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 1 -- 172.21.15.130:0/890303330 shutdown_connections 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 1 -- 172.21.15.130:0/890303330 >> 172.21.15.130:0/890303330 conn(0x7f001c0b7ea0 msgr2=0x7f001c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 1 -- 172.21.15.130:0/890303330 shutdown_connections 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 1 -- 172.21.15.130:0/890303330 wait complete. 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 10 monclient: build_initial_monmap 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.858+0000 7f00243a3640 10 monclient: monmap: 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:35.860 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 librados: starting msgr at 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 librados: starting objecter 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 Processor -- start 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 -- start start 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 librados: setting wanted keys 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 librados: calling monclient init 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient: init 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:35.861 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-a con 0x7f001c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-c con 0x7f001c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient(hunting): picked mon.noname-b con 0x7f001c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c138c10 con 0x7f001c0a98f0 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c138e60 con 0x7f001c1430c0 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f001c139140 con 0x7f001c13f710 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:35.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.859+0000 7f00243a3640 10 monclient(hunting): authenticate will time out at 5392.005533s 2024-03-25T19:48:35.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.867+0000 7f002213f640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f001c0a98f0 0x7f001c137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:59150/0 (socket says 172.21.15.130:59150) 2024-03-25T19:48:35.867 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.867+0000 7f002213f640 1 -- 172.21.15.130:0/2977167000 learned_addr learned my addr 172.21.15.130:0/2977167000 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:35.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.873+0000 7f0012ffd640 10 client.?.objecter ms_handle_connect 0x7f001c0a98f0 2024-03-25T19:48:35.872 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.873+0000 7f0012ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:35.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.881+0000 7f0012ffd640 10 client.?.objecter ms_handle_connect 0x7f001c1430c0 2024-03-25T19:48:35.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.881+0000 7f0012ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:35.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.882+0000 7f0012ffd640 10 client.?.objecter ms_handle_connect 0x7f001c13f710 2024-03-25T19:48:35.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.882+0000 7f0012ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627122201 0 0) 0x7f001c138c10 con 0x7f001c0a98f0 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 10 monclient(hunting): my global_id is 4542 2024-03-25T19:48:35.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.884+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0000004eb0 con 0x7f001c0a98f0 2024-03-25T19:48:35.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4025866123 0 0) 0x7f001c139140 con 0x7f001c13f710 2024-03-25T19:48:35.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 10 monclient(hunting): my global_id is 4535 2024-03-25T19:48:35.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.888+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c138c10 con 0x7f001c13f710 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552512564 0 0) 0x7f001c138e60 con 0x7f001c1430c0 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 10 monclient(hunting): my global_id is 4534 2024-03-25T19:48:35.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.892+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c139140 con 0x7f001c1430c0 2024-03-25T19:48:35.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872969559 0 0) 0x7f001c138c10 con 0x7f001c13f710 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f001c138e60 con 0x7f001c13f710 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0018001b00 con 0x7f001c13f710 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 10 monclient(hunting): dump: 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:35.896 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:35.897 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:35.897 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.897 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.896+0000 7f0012ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:35.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1969696051 0 0) 0x7f0000004eb0 con 0x7f001c0a98f0 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f001c138c10 con 0x7f001c0a98f0 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f000c001a30 con 0x7f001c0a98f0 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 10 monclient(hunting): dump: 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:35.904 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.903+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1258244722 0 0) 0x7f001c138e60 con 0x7f001c13f710 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 >> v1:172.21.15.163:6789/0 conn(0x7f001c1430c0 legacy=0x7f001c1e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 >> v1:172.21.15.130:6789/0 conn(0x7f001c0a98f0 legacy=0x7f001c137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 1 monclient: found mon.c 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c139920 con 0x7f001c13f710 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 10 monclient: _finish_auth 0 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f0012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 5 monclient: authenticate success, global_id 4535 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 10 monclient: _renew_subs 2024-03-25T19:48:35.905 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c1382e0 con 0x7f001c13f710 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 10 client.4535.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 10 monclient: _renew_subs 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c0ab8b0 con 0x7f001c13f710 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.904+0000 7f00243a3640 1 librados: init done 2024-03-25T19:48:35.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.905+0000 7f00243a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:35.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.905+0000 7f00243a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:35.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.905+0000 7f00243a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:35.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.905+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c13d010 con 0x7f001c13f710 2024-03-25T19:48:35.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.911+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0018001ca0 con 0x7f001c13f710 2024-03-25T19:48:35.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.911+0000 7f0012ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:35.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:35.911+0000 7f002093c640 4 set_mon_vals no callback set 2024-03-25T19:48:36.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.201+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f0018004c30 con 0x7f001c13f710 2024-03-25T19:48:36.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.202+0000 7f0012ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.202+0000 7f0012ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.202+0000 7f0012ffd640 10 monclient: dump: 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:36.202 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:36.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.202+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f001801ac00 con 0x7f001c13f710 2024-03-25T19:48:36.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.203+0000 7f002213f640 10 monclient: get_auth_request con 0x7f000006df70 auth_method 0 2024-03-25T19:48:36.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.203+0000 7f0012ffd640 10 client.4535.objecter ms_handle_connect 0x7f000006df70 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f00180548b0 con 0x7f001c13f710 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 10 client.4535.objecter ms_dispatch 0x7f001c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 3 client.4535.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 3 client.4535.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 20 client.4535.objecter dump_active .. 0 homeless 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f00180501f0 con 0x7f001c13f710 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:36.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.209+0000 7f0012ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:36.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.446+0000 7f00243a3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:36.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.446+0000 7f00243a3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:36.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.446+0000 7f00243a3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:36.447 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.446+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f001c139660 con 0x7f001c13f710 2024-03-25T19:48:36.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.535+0000 7f0012ffd640 1 -- 172.21.15.130:0/2977167000 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f001c139660 con 0x7f001c13f710 2024-03-25T19:48:36.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.535+0000 7f0012ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:36.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.535+0000 7f0012ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:36.535 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:36.535 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 10 librados: watch_flush enter 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 10 librados: watch_flush exit 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 20 client.4535.objecter shutdown clearing up homeless session... 2024-03-25T19:48:36.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 10 client.4535.objecter successfully canceled tick 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 >> v1:172.21.15.163:6800/1217380380 conn(0x7f000006df70 legacy=0x7f0000070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 10 monclient: shutdown 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 >> v1:172.21.15.130:6790/0 conn(0x7f001c13f710 legacy=0x7f001c132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 shutdown_connections 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 >> 172.21.15.130:0/2977167000 conn(0x7f001c0b7ea0 msgr2=0x7f001c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 shutdown_connections 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 -- 172.21.15.130:0/2977167000 wait complete. 2024-03-25T19:48:36.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:36.537+0000 7f00243a3640 1 librados: shutdown 2024-03-25T19:48:36.549 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:43.356 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.472+0000 7faf90396640 10 monclient: get_monmap_and_config 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.472+0000 7faf90396640 10 monclient: build_initial_monmap 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.472+0000 7faf90396640 10 monclient: monmap: 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:43.473573+0000 2024-03-25T19:48:43.472 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:43.473573+0000 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:43.473 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.473+0000 7faf90396640 1 Processor -- start 2024-03-25T19:48:43.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.473+0000 7faf90396640 1 -- start start 2024-03-25T19:48:43.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.473+0000 7faf90396640 10 monclient: init 2024-03-25T19:48:43.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:43.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient(hunting): picked mon.noname-a con 0x7faf880a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient(hunting): picked mon.noname-c con 0x7faf8813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient(hunting): picked mon.noname-b con 0x7faf881430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf88142520 con 0x7faf880a98f0 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf880b28a0 con 0x7faf8813f710 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf880b2a30 con 0x7faf881430c0 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:43.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.474+0000 7faf90396640 10 monclient(hunting): authenticate will time out at 5399.619819s 2024-03-25T19:48:43.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.488+0000 7faf8e933640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7faf881430c0 0x7faf8814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:46662/0 (socket says 172.21.15.130:46662) 2024-03-25T19:48:43.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.488+0000 7faf8e933640 1 -- 172.21.15.130:0/302780868 learned_addr learned my addr 172.21.15.130:0/302780868 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:43.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3187600734 0 0) 0x7faf880b2a30 con 0x7faf881430c0 2024-03-25T19:48:43.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 10 monclient(hunting): my global_id is 4538 2024-03-25T19:48:43.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.500+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf74003520 con 0x7faf881430c0 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184747365 0 0) 0x7faf880b28a0 con 0x7faf8813f710 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 10 monclient(hunting): my global_id is 4537 2024-03-25T19:48:43.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.502+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf880b2a30 con 0x7faf8813f710 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746667992 0 0) 0x7faf88142520 con 0x7faf880a98f0 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 10 monclient(hunting): my global_id is 4545 2024-03-25T19:48:43.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.503+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf880b28a0 con 0x7faf880a98f0 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 208790872 0 0) 0x7faf880b28a0 con 0x7faf880a98f0 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 >> v1:172.21.15.130:6790/0 conn(0x7faf881430c0 legacy=0x7faf8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 >> v1:172.21.15.163:6789/0 conn(0x7faf8813f710 legacy=0x7faf881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: _finish_hunting 0 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 monclient: found mon.noname-a 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf880b44c0 con 0x7faf880a98f0 2024-03-25T19:48:43.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: _finish_auth 0 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: _check_auth_tickets 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf840015b0 con 0x7faf880a98f0 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf90396640 5 monclient: authenticate success, global_id 4545 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf90396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf8d130640 10 monclient: dump: 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.521 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.522 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.520+0000 7faf90396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:43.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf84001710 con 0x7faf880a98f0 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 1 -- 172.21.15.130:0/302780868 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf84001b60 con 0x7faf880a98f0 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf8d130640 10 monclient: dump: 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.563 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.564 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf90396640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:43.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf90396640 4 set_mon_vals no callback set 2024-03-25T19:48:43.565 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf90396640 10 monclient: shutdown 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf90396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.563+0000 7faf90396640 1 -- 172.21.15.130:0/302780868 >> v1:172.21.15.130:6789/0 conn(0x7faf880a98f0 legacy=0x7faf880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 1 -- 172.21.15.130:0/302780868 shutdown_connections 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 1 -- 172.21.15.130:0/302780868 >> 172.21.15.130:0/302780868 conn(0x7faf880b7ea0 msgr2=0x7faf880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 1 -- 172.21.15.130:0/302780868 shutdown_connections 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 1 -- 172.21.15.130:0/302780868 wait complete. 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 10 monclient: build_initial_monmap 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.564+0000 7faf90396640 10 monclient: monmap: 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:43.566 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 librados: starting msgr at 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 librados: starting objecter 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 Processor -- start 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 -- start start 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 librados: setting wanted keys 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 librados: calling monclient init 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient: init 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient(hunting): picked mon.noname-c con 0x7faf880a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient(hunting): picked mon.noname-a con 0x7faf8813f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient(hunting): picked mon.noname-b con 0x7faf881430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:43.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf88136830 con 0x7faf8813f710 2024-03-25T19:48:43.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf88137e40 con 0x7faf881430c0 2024-03-25T19:48:43.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf88138120 con 0x7faf880a98f0 2024-03-25T19:48:43.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:43.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.565+0000 7faf90396640 10 monclient(hunting): authenticate will time out at 5399.711523s 2024-03-25T19:48:43.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.573+0000 7faf8d931640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7faf8813f710 0x7faf88138590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43982/0 (socket says 172.21.15.130:43982) 2024-03-25T19:48:43.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.574+0000 7faf8d931640 1 -- 172.21.15.130:0/3262422654 learned_addr learned my addr 172.21.15.130:0/3262422654 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:43.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.576+0000 7faf7effd640 10 client.?.objecter ms_handle_connect 0x7faf8813f710 2024-03-25T19:48:43.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.576+0000 7faf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:43.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.583+0000 7faf7effd640 10 client.?.objecter ms_handle_connect 0x7faf880a98f0 2024-03-25T19:48:43.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.583+0000 7faf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:43.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.588+0000 7faf7effd640 10 client.?.objecter ms_handle_connect 0x7faf881430c0 2024-03-25T19:48:43.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.588+0000 7faf7effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2395144999 0 0) 0x7faf88138120 con 0x7faf880a98f0 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 10 monclient(hunting): my global_id is 4541 2024-03-25T19:48:43.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.592+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf6c004eb0 con 0x7faf880a98f0 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1816908541 0 0) 0x7faf88136830 con 0x7faf8813f710 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 10 monclient(hunting): my global_id is 4548 2024-03-25T19:48:43.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.597+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf88138120 con 0x7faf8813f710 2024-03-25T19:48:43.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270343173 0 0) 0x7faf88137e40 con 0x7faf881430c0 2024-03-25T19:48:43.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 10 monclient(hunting): _negotiate 2024-03-25T19:48:43.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:43.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:43.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 10 monclient(hunting): my global_id is 4540 2024-03-25T19:48:43.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.604+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf88136830 con 0x7faf881430c0 2024-03-25T19:48:43.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1712168280 0 0) 0x7faf6c004eb0 con 0x7faf880a98f0 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf88137e40 con 0x7faf880a98f0 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf84001a20 con 0x7faf880a98f0 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 10 monclient(hunting): dump: 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.609 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.610 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.610 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.610 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.609+0000 7faf7effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:43.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3804016722 0 0) 0x7faf88136830 con 0x7faf881430c0 2024-03-25T19:48:43.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf6c004eb0 con 0x7faf881430c0 2024-03-25T19:48:43.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf80001a30 con 0x7faf881430c0 2024-03-25T19:48:43.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.613+0000 7faf7effd640 10 monclient(hunting): dump: 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.614 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4183933660 0 0) 0x7faf88138120 con 0x7faf8813f710 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf88136830 con 0x7faf8813f710 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf780019f0 con 0x7faf8813f710 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.614+0000 7faf7effd640 10 monclient(hunting): dump: 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.615 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.616 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.616 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.616 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.616 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.616 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3229743839 0 0) 0x7faf88137e40 con 0x7faf880a98f0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 >> v1:172.21.15.163:6789/0 conn(0x7faf881430c0 legacy=0x7faf881e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 >> v1:172.21.15.130:6789/0 conn(0x7faf8813f710 legacy=0x7faf88138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 10 monclient: _finish_hunting 0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 1 monclient: found mon.c 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf880abac0 con 0x7faf880a98f0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 10 monclient: _finish_auth 0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 10 monclient: _check_auth_tickets 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.620+0000 7faf90396640 5 monclient: authenticate success, global_id 4541 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 10 monclient: _renew_subs 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:43.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf881db7f0 con 0x7faf880a98f0 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 10 monclient: _renew_subs 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf88137a10 con 0x7faf880a98f0 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.621+0000 7faf90396640 1 librados: init done 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.622+0000 7faf90396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:43.622 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.622+0000 7faf90396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:43.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.622+0000 7faf90396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:43.623 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.622+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf880627b0 con 0x7faf880a98f0 2024-03-25T19:48:43.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faf840038a0 con 0x7faf880a98f0 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7faf84003bb0 con 0x7faf880a98f0 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 10 monclient: dump: 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:43.928 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf8c92f640 4 set_mon_vals no callback set 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.928+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7faf84019b80 con 0x7faf880a98f0 2024-03-25T19:48:43.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.929+0000 7faf8d931640 10 monclient: get_auth_request con 0x7faf6c06e0b0 auth_method 0 2024-03-25T19:48:43.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.929+0000 7faf7effd640 10 client.4541.objecter ms_handle_connect 0x7faf6c06e0b0 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7faf840536c0 con 0x7faf880a98f0 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 10 client.4541.objecter ms_dispatch 0x7faf88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faf8404f1d0 con 0x7faf880a98f0 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:43.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:43.935+0000 7faf7effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:44.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.189+0000 7faf90396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.189+0000 7faf90396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.189+0000 7faf90396640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:48:44.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.189+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf88069f40 con 0x7faf880a98f0 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.215+0000 7faf7effd640 1 -- 172.21.15.130:0/3262422654 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7faf8404f1d0 con 0x7faf880a98f0 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.215+0000 7faf7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:44.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.215+0000 7faf7effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:44.215 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:44.215 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.217+0000 7faf90396640 10 librados: watch_flush enter 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.217+0000 7faf90396640 10 librados: watch_flush exit 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.217+0000 7faf90396640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.217+0000 7faf90396640 10 client.4541.objecter successfully canceled tick 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 >> v1:172.21.15.163:6800/1217380380 conn(0x7faf6c06e0b0 legacy=0x7faf6c070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 10 monclient: shutdown 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 >> v1:172.21.15.130:6790/0 conn(0x7faf880a98f0 legacy=0x7faf881db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 shutdown_connections 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 >> 172.21.15.130:0/3262422654 conn(0x7faf880b7ea0 msgr2=0x7faf8813d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:44.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 shutdown_connections 2024-03-25T19:48:44.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 -- 172.21.15.130:0/3262422654 wait complete. 2024-03-25T19:48:44.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:44.218+0000 7faf90396640 1 librados: shutdown 2024-03-25T19:48:44.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:45.729 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:30.662 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.741 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:30.674 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.774 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:07:30.707 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.787 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:30.688 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.792 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:30.725 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.795 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:30.696 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.800 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:30.733 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.826 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:07:30.727 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:45.836 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:07:30.769 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:48:51.036 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:51.153 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.154+0000 7fc9e1216640 10 monclient: get_monmap_and_config 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.154+0000 7fc9e1216640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.154+0000 7fc9e1216640 10 monclient: monmap: 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:51.155160+0000 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:51.155160+0000 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:51.154 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.155+0000 7fc9e1216640 1 Processor -- start 2024-03-25T19:48:51.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.155+0000 7fc9e1216640 1 -- start start 2024-03-25T19:48:51.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.155+0000 7fc9e1216640 10 monclient: init 2024-03-25T19:48:51.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:48:51.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-a con 0x7fc9dc0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:51.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-b con 0x7fc9dc0b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:51.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-c con 0x7fc9dc14be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0af940 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0b0a70 con 0x7fc9dc14be90 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0b0d50 con 0x7fc9dc0b1730 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.156+0000 7fc9e1216640 10 monclient(hunting): authenticate will time out at 5407.302279s 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.158+0000 7fc9db59e640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc9dc14be90 0x7fc9dc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:54696/0 (socket says 172.21.15.130:54696) 2024-03-25T19:48:51.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.158+0000 7fc9db59e640 1 -- 172.21.15.130:0/2945965440 learned_addr learned my addr 172.21.15.130:0/2945965440 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:51.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596032722 0 0) 0x7fc9dc0af940 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 10 monclient(hunting): my global_id is 4551 2024-03-25T19:48:51.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.236+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c8003520 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.247+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265483347 0 0) 0x7fc9dc0b0a70 con 0x7fc9dc14be90 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.247+0000 7fc9da59c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.247+0000 7fc9da59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.247+0000 7fc9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 10 monclient(hunting): my global_id is 4543 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc0af940 con 0x7fc9dc14be90 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776617252 0 0) 0x7fc9dc0b0d50 con 0x7fc9dc0b1730 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 10 monclient(hunting): my global_id is 4544 2024-03-25T19:48:51.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.248+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc0b0a70 con 0x7fc9dc0b1730 2024-03-25T19:48:51.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3075861614 0 0) 0x7fc9dc0af940 con 0x7fc9dc14be90 2024-03-25T19:48:51.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 >> v1:172.21.15.130:6790/0 conn(0x7fc9dc0b1730 legacy=0x7fc9dc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 >> v1:172.21.15.130:6789/0 conn(0x7fc9dc0b4d60 legacy=0x7fc9dc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 monclient: found mon.noname-c 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc0b1030 con 0x7fc9dc14be90 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc9d40019f0 con 0x7fc9dc14be90 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9e1216640 5 monclient: authenticate success, global_id 4543 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9e1216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.254+0000 7fc9da59c640 10 monclient: dump: 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.255+0000 7fc9e1216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d4001b50 con 0x7fc9dc14be90 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 1 -- 172.21.15.130:0/2945965440 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc9d4001f60 con 0x7fc9dc14be90 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9da59c640 10 monclient: dump: 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:51.282 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9e1216640 10 monclient: get_monmap_and_config success 2024-03-25T19:48:51.283 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.282+0000 7fc9e1216640 4 set_mon_vals no callback set 2024-03-25T19:48:51.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 10 monclient: shutdown 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 1 -- 172.21.15.130:0/2945965440 >> v1:172.21.15.163:6789/0 conn(0x7fc9dc14be90 legacy=0x7fc9dc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 1 -- 172.21.15.130:0/2945965440 shutdown_connections 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 1 -- 172.21.15.130:0/2945965440 >> 172.21.15.130:0/2945965440 conn(0x7fc9dc0617a0 msgr2=0x7fc9dc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 1 -- 172.21.15.130:0/2945965440 shutdown_connections 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.283+0000 7fc9e1216640 1 -- 172.21.15.130:0/2945965440 wait complete. 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.284+0000 7fc9e1216640 10 monclient: build_initial_monmap 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.284+0000 7fc9e1216640 10 monclient: monmap: 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:51.285 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 librados: starting msgr at 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 librados: starting objecter 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 Processor -- start 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 -- start start 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 librados: setting wanted keys 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 1 librados: calling monclient init 2024-03-25T19:48:51.286 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 10 monclient: init 2024-03-25T19:48:51.287 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.285+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-a con 0x7fc9dc0b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-b con 0x7fc9dc0b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 10 monclient(hunting): picked mon.noname-c con 0x7fc9dc14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0aabb0 con 0x7fc9dc0b1730 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc0a95a0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9dc13b540 con 0x7fc9dc14be90 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:51.288 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.286+0000 7fc9e1216640 10 monclient(hunting): authenticate will time out at 5407.431827s 2024-03-25T19:48:51.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.292+0000 7fc9d3fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fc9dc0b4d60 0x7fc9dc1d7a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:54748/0 (socket says 172.21.15.130:54748) 2024-03-25T19:48:51.292 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.293+0000 7fc9d3fff640 1 -- 172.21.15.130:0/857638317 learned_addr learned my addr 172.21.15.130:0/857638317 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:48:51.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.298+0000 7fc9d37fe640 10 client.?.objecter ms_handle_connect 0x7fc9dc0b4d60 2024-03-25T19:48:51.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.298+0000 7fc9d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.308 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548165156 0 0) 0x7fc9dc0a95a0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 10 monclient(hunting): my global_id is 4546 2024-03-25T19:48:51.309 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.309+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc004eb0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.319+0000 7fc9d37fe640 10 client.?.objecter ms_handle_connect 0x7fc9dc0b1730 2024-03-25T19:48:51.319 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.319+0000 7fc9d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.323+0000 7fc9d37fe640 10 client.?.objecter ms_handle_connect 0x7fc9dc14be90 2024-03-25T19:48:51.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.323+0000 7fc9d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:48:51.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3690163587 0 0) 0x7fc9bc004eb0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9dc0a95a0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc9c4001a10 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 10 monclient(hunting): dump: 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.327 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.326+0000 7fc9d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:48:51.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3267825727 0 0) 0x7fc9dc13b540 con 0x7fc9dc14be90 2024-03-25T19:48:51.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): my global_id is 4547 2024-03-25T19:48:51.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc004eb0 con 0x7fc9dc14be90 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726948603 0 0) 0x7fc9dc0aabb0 con 0x7fc9dc0b1730 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 10 monclient(hunting): my global_id is 4554 2024-03-25T19:48:51.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.330+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9dc13b540 con 0x7fc9dc0b1730 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.337+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1248995701 0 0) 0x7fc9dc0a95a0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 >> v1:172.21.15.130:6790/0 conn(0x7fc9dc14be90 legacy=0x7fc9dc1d8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 >> v1:172.21.15.130:6789/0 conn(0x7fc9dc0b1730 legacy=0x7fc9dc137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 1 monclient: found mon.b 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:48:51.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9dc13b6d0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 10 monclient: _finish_auth 0 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 5 monclient: authenticate success, global_id 4546 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 10 monclient: _renew_subs 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9dc138680 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 10 monclient: _renew_subs 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9dc138f40 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.338+0000 7fc9e1216640 1 librados: init done 2024-03-25T19:48:51.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.339+0000 7fc9e1216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.339+0000 7fc9e1216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.339+0000 7fc9e1216640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.340 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.339+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc138980 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.348 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.349+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9c4001cb0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.349+0000 7fc9d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:48:51.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.349+0000 7fc9d9d9b640 4 set_mon_vals no callback set 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.625+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc9c4003a60 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.625+0000 7fc9d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.625+0000 7fc9d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.625+0000 7fc9d37fe640 10 monclient: dump: 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:48:51.625 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.626+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fc9c4003c40 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.626+0000 7fc9dad9d640 10 monclient: get_auth_request con 0x7fc9bc06de80 auth_method 0 2024-03-25T19:48:51.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.627+0000 7fc9d37fe640 10 client.4546.objecter ms_handle_connect 0x7fc9bc06de80 2024-03-25T19:48:51.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.629+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fc9c40548b0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.629+0000 7fc9d37fe640 10 client.4546.objecter ms_dispatch 0x7fc9dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:48:51.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.629+0000 7fc9d37fe640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:48:51.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.629+0000 7fc9d37fe640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:48:51.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.629+0000 7fc9d37fe640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T19:48:51.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.632+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9c40501f0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.632+0000 7fc9d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:48:51.631 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.632+0000 7fc9d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:48:51.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.881+0000 7fc9e1216640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:51.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.881+0000 7fc9e1216640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:51.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.881+0000 7fc9e1216640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:48:51.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.881+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9dc0afbd0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.899+0000 7fc9d37fe640 1 -- 172.21.15.130:0/857638317 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fc9c40522f0 con 0x7fc9dc0b4d60 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.900+0000 7fc9d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:48:51.899 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.900+0000 7fc9d37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:48:51.900 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:48:51.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 10 librados: watch_flush enter 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 10 librados: watch_flush exit 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 10 client.4546.objecter successfully canceled tick 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 >> v1:172.21.15.163:6800/1217380380 conn(0x7fc9bc06de80 legacy=0x7fc9bc070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 10 monclient: shutdown 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 >> v1:172.21.15.163:6789/0 conn(0x7fc9dc0b4d60 legacy=0x7fc9dc1d7a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 shutdown_connections 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.902+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 >> 172.21.15.130:0/857638317 conn(0x7fc9dc0617a0 msgr2=0x7fc9dc13beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.903+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 shutdown_connections 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.903+0000 7fc9e1216640 1 -- 172.21.15.130:0/857638317 wait complete. 2024-03-25T19:48:51.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:51.903+0000 7fc9e1216640 1 librados: shutdown 2024-03-25T19:48:51.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:48:58.722 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 10 monclient: get_monmap_and_config 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 10 monclient: build_initial_monmap 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 10 monclient: monmap: 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:48:58.812012+0000 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:48:58.812012+0000 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:48:58.811 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 1 Processor -- start 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 1 -- start start 2024-03-25T19:48:58.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.810+0000 7f2364f7b640 10 monclient: init 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f23600b4f40 addr v1:172.21.15.130:6790/0 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f236013fd90 addr v1:172.21.15.163:6789/0 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f23600b4540 addr v1:172.21.15.130:6789/0 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600aa0c0 con 0x7f23600b4540 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600ab170 con 0x7f236013fd90 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600a9ba0 con 0x7f23600b4f40 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:48:58.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.811+0000 7f2364f7b640 10 monclient(hunting): authenticate will time out at 5414.958153s 2024-03-25T19:48:58.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.812+0000 7f235e59c640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f23600b4f40 0x7f236013f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:44858/0 (socket says 172.21.15.130:44858) 2024-03-25T19:48:58.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:48:58.812+0000 7f235e59c640 1 -- 172.21.15.130:0/2236030777 learned_addr learned my addr 172.21.15.130:0/2236030777 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:01.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): tick 2024-03-25T19:49:01.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:49:01.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6790/0 conn(0x7f23600b4f40 legacy=0x7f236013f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.163:6789/0 conn(0x7f236013fd90 legacy=0x7f23601423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6789/0 conn(0x7f23600b4540 legacy=0x7f236014dc30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f23400036c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f236013fd90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f23600b4f40 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600a9680 con 0x7f23400036c0 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600aa0c0 con 0x7f236013fd90 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23400103b0 con 0x7f23600b4f40 2024-03-25T19:49:01.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:01.811+0000 7f234ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:04.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): tick 2024-03-25T19:49:04.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6790/0 conn(0x7f23600b4f40 legacy=0x7f234000cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.163:6789/0 conn(0x7f236013fd90 legacy=0x7f2340009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6789/0 conn(0x7f23400036c0 legacy=0x7f2340005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f234000f4e0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f236013fd90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f23600b4f40 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340011610 con 0x7f234000f4e0 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600a9680 con 0x7f236013fd90 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340004ab0 con 0x7f23600b4f40 2024-03-25T19:49:04.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:04.811+0000 7f234ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): tick 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): _check_auth_tickets 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 monclient(hunting): continuing hunt 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6790/0 conn(0x7f23600b4f40 legacy=0x7f234000cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.163:6789/0 conn(0x7f236013fd90 legacy=0x7f2340009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6789/0 conn(0x7f234000f4e0 legacy=0x7f234000f920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:07.813 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-c con 0x7f2340003c60 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-b con 0x7f23600b4f40 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): picked mon.noname-a con 0x7f234000f4e0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340005ca0 con 0x7f234000f4e0 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340011610 con 0x7f2340003c60 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340005e50 con 0x7f23600b4f40 2024-03-25T19:49:07.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:07.812+0000 7f234ffff640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581735778 0 0) 0x7f2340005e50 con 0x7f23600b4f40 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 10 monclient(hunting): my global_id is 4550 2024-03-25T19:49:08.723 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.722+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2348003520 con 0x7f23600b4f40 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3987452670 0 0) 0x7f2348003520 con 0x7f23600b4f40 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.163:6789/0 conn(0x7f2340003c60 legacy=0x7f23400040a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6789/0 conn(0x7f234000f4e0 legacy=0x7f234000f920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 1 monclient: found mon.noname-b 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.727+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2340006ff0 con 0x7f23600b4f40 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2354001670 con 0x7f23600b4f40 2024-03-25T19:49:08.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f2364f7b640 5 monclient: authenticate success, global_id 4550 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f2364f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f235d59a640 10 monclient: dump: 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:08.730 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:08.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.728+0000 7f2364f7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:08.753 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23540017d0 con 0x7f23600b4f40 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 1 -- 172.21.15.130:0/2236030777 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2354001c20 con 0x7f23600b4f40 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f235d59a640 10 monclient: dump: 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.754 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f2364f7b640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:08.755 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.753+0000 7f2364f7b640 4 set_mon_vals no callback set 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 10 monclient: shutdown 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 1 -- 172.21.15.130:0/2236030777 >> v1:172.21.15.130:6790/0 conn(0x7f23600b4f40 legacy=0x7f234000cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 1 -- 172.21.15.130:0/2236030777 shutdown_connections 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 1 -- 172.21.15.130:0/2236030777 >> 172.21.15.130:0/2236030777 conn(0x7f23600617a0 msgr2=0x7f2360061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 1 -- 172.21.15.130:0/2236030777 shutdown_connections 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.754+0000 7f2364f7b640 1 -- 172.21.15.130:0/2236030777 wait complete. 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.755+0000 7f2364f7b640 10 monclient: build_initial_monmap 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.755+0000 7f2364f7b640 10 monclient: monmap: 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:08.756 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:08.757 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:08.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 librados: starting msgr at 2024-03-25T19:49:08.758 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 librados: starting objecter 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 Processor -- start 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 -- start start 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 librados: setting wanted keys 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 1 librados: calling monclient init 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.756+0000 7f2364f7b640 10 monclient: init 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-c con 0x7f2340003c60 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-a con 0x7f234000f4e0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient(hunting): picked mon.noname-b con 0x7f23600b4f40 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23601d8610 con 0x7f234000f4e0 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600b0bb0 con 0x7f23600b4f40 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23600b0e90 con 0x7f2340003c60 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:08.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.757+0000 7f2364f7b640 10 monclient(hunting): authenticate will time out at 5424.903787s 2024-03-25T19:49:08.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.769+0000 7f235e59c640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f2340003c60 0x7f23601d7f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:42040/0 (socket says 172.21.15.130:42040) 2024-03-25T19:49:08.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.769+0000 7f235e59c640 1 -- 172.21.15.130:0/3950011962 learned_addr learned my addr 172.21.15.130:0/3950011962 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:08.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.770+0000 7f234effd640 10 client.?.objecter ms_handle_connect 0x7f2340003c60 2024-03-25T19:49:08.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.770+0000 7f234effd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:08.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164012801 0 0) 0x7f23600b0e90 con 0x7f2340003c60 2024-03-25T19:49:08.783 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:08.783 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:08.783 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:08.783 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 10 monclient(hunting): my global_id is 4553 2024-03-25T19:49:08.783 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.782+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234000f8c0 con 0x7f2340003c60 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1843022652 0 0) 0x7f234000f8c0 con 0x7f2340003c60 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23600b0e90 con 0x7f2340003c60 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f23500016e0 con 0x7f2340003c60 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 10 monclient(hunting): dump: 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:08.800 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:08.801 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.799+0000 7f234effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:08.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1318299655 0 0) 0x7f23600b0e90 con 0x7f2340003c60 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 >> v1:172.21.15.163:6789/0 conn(0x7f23600b4f40 legacy=0x7f23601e3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 >> v1:172.21.15.130:6789/0 conn(0x7f234000f4e0 legacy=0x7f2360137590 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 1 monclient: found mon.c 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23601d79e0 con 0x7f2340003c60 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 10 monclient: _finish_auth 0 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f234effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 5 monclient: authenticate success, global_id 4553 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 10 monclient: _renew_subs 2024-03-25T19:49:08.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2360138290 con 0x7f2340003c60 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 10 monclient: _renew_subs 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23600b1900 con 0x7f2340003c60 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.827+0000 7f2364f7b640 1 librados: init done 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.829+0000 7f2364f7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.829+0000 7f2364f7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.829+0000 7f2364f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:49:08.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:08.829+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23601d77d0 con 0x7f2340003c60 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2350001c30 con 0x7f2340003c60 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2350004980 con 0x7f2340003c60 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 10 monclient: dump: 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:09.126 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f235cd99640 4 set_mon_vals no callback set 2024-03-25T19:49:09.127 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.125+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f2350004ba0 con 0x7f2340003c60 2024-03-25T19:49:09.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.126+0000 7f235dd9b640 10 monclient: get_auth_request con 0x7f234006f680 auth_method 0 2024-03-25T19:49:09.128 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.126+0000 7f234effd640 10 client.4553.objecter ms_handle_connect 0x7f234006f680 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f2350054720 con 0x7f2340003c60 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 10 client.4553.objecter ms_dispatch 0x7f2360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2350050230 con 0x7f2340003c60 2024-03-25T19:49:09.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:09.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.131+0000 7f234effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.325+0000 7f2364f7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.325+0000 7f2364f7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.325+0000 7f2364f7b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:49:09.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.325+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23600627b0 con 0x7f2340003c60 2024-03-25T19:49:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.411+0000 7f234effd640 1 -- 172.21.15.130:0/3950011962 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f2350052160 con 0x7f2340003c60 2024-03-25T19:49:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.411+0000 7f234effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:09.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.411+0000 7f234effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:09.412 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:09.412 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 10 librados: watch_flush enter 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 10 librados: watch_flush exit 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 10 client.4553.objecter successfully canceled tick 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 >> v1:172.21.15.163:6800/1217380380 conn(0x7f234006f680 legacy=0x7f2340071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 10 monclient: shutdown 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:09.415 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 >> v1:172.21.15.130:6790/0 conn(0x7f2340003c60 legacy=0x7f23601d7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:09.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 shutdown_connections 2024-03-25T19:49:09.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 >> 172.21.15.130:0/3950011962 conn(0x7f23600617a0 msgr2=0x7f23600b8580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:09.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 shutdown_connections 2024-03-25T19:49:09.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 -- 172.21.15.130:0/3950011962 wait complete. 2024-03-25T19:49:09.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:09.414+0000 7f2364f7b640 1 librados: shutdown 2024-03-25T19:49:09.427 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:16.232 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:16.344 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 10 monclient: get_monmap_and_config 2024-03-25T19:49:16.344 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 10 monclient: monmap: 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:16.345993+0000 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:16.345993+0000 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:16.345 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 1 Processor -- start 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 1 -- start start 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.345+0000 7fcdf4332640 10 monclient: init 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-a con 0x7fcdec0adaf0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-b con 0x7fcdec13fd20 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-c con 0x7fcdec142a90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec06fd30 con 0x7fcdec0adaf0 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec06e920 con 0x7fcdec142a90 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec0b01e0 con 0x7fcdec13fd20 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.346+0000 7fcdf4332640 10 monclient(hunting): authenticate will time out at 5432.492143s 2024-03-25T19:49:16.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.347+0000 7fcdf20ce640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fcdec0adaf0 0x7fcdec13f610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:51622/0 (socket says 172.21.15.130:51622) 2024-03-25T19:49:16.348 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.347+0000 7fcdf20ce640 1 -- 172.21.15.130:0/10874720 learned_addr learned my addr 172.21.15.130:0/10874720 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:16.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062723637 0 0) 0x7fcdec06e920 con 0x7fcdec142a90 2024-03-25T19:49:16.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 10 monclient(hunting): my global_id is 4552 2024-03-25T19:49:16.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.474+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd8003520 con 0x7fcdec142a90 2024-03-25T19:49:16.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2362964350 0 0) 0x7fcdd8003520 con 0x7fcdec142a90 2024-03-25T19:49:16.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 >> v1:172.21.15.130:6790/0 conn(0x7fcdec13fd20 legacy=0x7fcdec142380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.486 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 >> v1:172.21.15.130:6789/0 conn(0x7fcdec0adaf0 legacy=0x7fcdec13f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 monclient: found mon.noname-c 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec0b0630 con 0x7fcdec142a90 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcde4001a10 con 0x7fcdec142a90 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf10cc640 10 monclient: dump: 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:16.487 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf4332640 5 monclient: authenticate success, global_id 4552 2024-03-25T19:49:16.488 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.486+0000 7fcdf4332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:16.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde4001b70 con 0x7fcdec142a90 2024-03-25T19:49:16.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 1 -- 172.21.15.130:0/10874720 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcde4001f00 con 0x7fcdec142a90 2024-03-25T19:49:16.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf10cc640 10 monclient: dump: 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf4332640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:16.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf4332640 4 set_mon_vals no callback set 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf4332640 10 monclient: shutdown 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf4332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.505+0000 7fcdf4332640 1 -- 172.21.15.130:0/10874720 >> v1:172.21.15.163:6789/0 conn(0x7fcdec142a90 legacy=0x7fcdec14dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 1 -- 172.21.15.130:0/10874720 shutdown_connections 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 1 -- 172.21.15.130:0/10874720 >> 172.21.15.130:0/10874720 conn(0x7fcdec0b0ad0 msgr2=0x7fcdec0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 1 -- 172.21.15.130:0/10874720 shutdown_connections 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 1 -- 172.21.15.130:0/10874720 wait complete. 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 10 monclient: build_initial_monmap 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.506+0000 7fcdf4332640 10 monclient: monmap: 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:16.507 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 librados: starting msgr at 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 librados: starting objecter 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 Processor -- start 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 -- start start 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 librados: setting wanted keys 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 1 librados: calling monclient init 2024-03-25T19:49:16.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.507+0000 7fcdf4332640 10 monclient: init 2024-03-25T19:49:16.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:16.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:16.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-a con 0x7fcdec0adaf0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-c con 0x7fcdec13fd20 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient(hunting): picked mon.noname-b con 0x7fcdec142a90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec11c710 con 0x7fcdec0adaf0 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec11c2f0 con 0x7fcdec142a90 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdec11c480 con 0x7fcdec13fd20 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:16.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.508+0000 7fcdf4332640 10 monclient(hunting): authenticate will time out at 5432.654271s 2024-03-25T19:49:16.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.512+0000 7fcdf28cf640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fcdec142a90 0x7fcdec1e2dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:46374/0 (socket says 172.21.15.130:46374) 2024-03-25T19:49:16.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.512+0000 7fcdf28cf640 1 -- 172.21.15.130:0/3717092897 learned_addr learned my addr 172.21.15.130:0/3717092897 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:16.514 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.514+0000 7fcde2ffd640 10 client.?.objecter ms_handle_connect 0x7fcdec142a90 2024-03-25T19:49:16.514 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.514+0000 7fcde2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673357904 0 0) 0x7fcdec11c2f0 con 0x7fcdec142a90 2024-03-25T19:49:16.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:16.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:16.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:16.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 10 monclient(hunting): my global_id is 4555 2024-03-25T19:49:16.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.526+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd0004eb0 con 0x7fcdec142a90 2024-03-25T19:49:16.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.529+0000 7fcde2ffd640 10 client.?.objecter ms_handle_connect 0x7fcdec0adaf0 2024-03-25T19:49:16.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.529+0000 7fcde2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:16.552 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3209859683 0 0) 0x7fcdd0004eb0 con 0x7fcdec142a90 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcdec11c2f0 con 0x7fcdec142a90 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcde4001ec0 con 0x7fcdec142a90 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 10 monclient(hunting): dump: 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:16.553 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:16.554 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.554 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.553+0000 7fcde2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1616088519 0 0) 0x7fcdec11c2f0 con 0x7fcdec142a90 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 >> v1:172.21.15.130:6790/0 conn(0x7fcdec13fd20 legacy=0x7fcdec11cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 >> v1:172.21.15.130:6789/0 conn(0x7fcdec0adaf0 legacy=0x7fcdec138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 1 monclient: found mon.b 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec1db750 con 0x7fcdec142a90 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcde2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 5 monclient: authenticate success, global_id 4555 2024-03-25T19:49:16.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 10 monclient: _renew_subs 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdec1db9c0 con 0x7fcdec142a90 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 10 client.4555.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 10 monclient: _renew_subs 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdec11bc10 con 0x7fcdec142a90 2024-03-25T19:49:16.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.572+0000 7fcdf4332640 1 librados: init done 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.573+0000 7fcdf4332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.573+0000 7fcdf4332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.573+0000 7fcdf4332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:16.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.573+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdec074d20 con 0x7fcdec142a90 2024-03-25T19:49:16.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcde4003c20 con 0x7fcdec142a90 2024-03-25T19:49:16.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fcde4003e50 con 0x7fcdec142a90 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcde2ffd640 10 monclient: dump: 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:16.829 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:16.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.828+0000 7fcdf08cb640 4 set_mon_vals no callback set 2024-03-25T19:49:16.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.830+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fcde4004070 con 0x7fcdec142a90 2024-03-25T19:49:16.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.830+0000 7fcdf20ce640 10 monclient: get_auth_request con 0x7fcdd006dc70 auth_method 0 2024-03-25T19:49:16.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.830+0000 7fcde2ffd640 10 client.4555.objecter ms_handle_connect 0x7fcdd006dc70 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fcde4054a30 con 0x7fcdec142a90 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 10 client.4555.objecter ms_dispatch 0x7fcdec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 3 client.4555.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 3 client.4555.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 20 client.4555.objecter dump_active .. 0 homeless 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcde4050540 con 0x7fcdec142a90 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:16.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:16.835+0000 7fcde2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:17.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.055+0000 7fcdf4332640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.055+0000 7fcdf4332640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.055+0000 7fcdf4332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:17.055 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.055+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcdec06e000 con 0x7fcdec142a90 2024-03-25T19:49:17.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.156+0000 7fcde2ffd640 1 -- 172.21.15.130:0/3717092897 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fcdec06e000 con 0x7fcdec142a90 2024-03-25T19:49:17.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.157+0000 7fcde2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:17.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.157+0000 7fcde2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:17.157 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:17.157 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:17.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 10 librados: watch_flush enter 2024-03-25T19:49:17.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 10 librados: watch_flush exit 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 20 client.4555.objecter shutdown clearing up homeless session... 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 10 client.4555.objecter successfully canceled tick 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 >> v1:172.21.15.163:6800/1217380380 conn(0x7fcdd006dc70 legacy=0x7fcdd0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 10 monclient: shutdown 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 >> v1:172.21.15.163:6789/0 conn(0x7fcdec142a90 legacy=0x7fcdec1e2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 shutdown_connections 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.159+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 >> 172.21.15.130:0/3717092897 conn(0x7fcdec0b0ad0 msgr2=0x7fcdec0b0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.160+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 shutdown_connections 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.160+0000 7fcdf4332640 1 -- 172.21.15.130:0/3717092897 wait complete. 2024-03-25T19:49:17.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:17.160+0000 7fcdf4332640 1 librados: shutdown 2024-03-25T19:49:17.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:23.978 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:24.092 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 10 monclient: get_monmap_and_config 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 10 monclient: monmap: 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:24.093911+0000 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:24.093911+0000 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:24.093 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 1 Processor -- start 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 1 -- start start 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.093+0000 7f1ced13d640 10 monclient: init 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce80b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80af940 con 0x7f1ce814be90 2024-03-25T19:49:24.095 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b0a70 con 0x7f1ce80b4d60 2024-03-25T19:49:24.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce80b0d50 con 0x7f1ce80b1730 2024-03-25T19:49:24.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ced13d640 10 monclient(hunting): authenticate will time out at 5440.239939s 2024-03-25T19:49:24.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ce6d9d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f1ce80b4d60 0x7f1ce813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:39662/0 (socket says 172.21.15.130:39662) 2024-03-25T19:49:24.096 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.094+0000 7f1ce6d9d640 1 -- 172.21.15.130:0/1061135679 learned_addr learned my addr 172.21.15.130:0/1061135679 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:24.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091045240 0 0) 0x7f1ce80af940 con 0x7f1ce814be90 2024-03-25T19:49:24.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 10 monclient(hunting): my global_id is 4560 2024-03-25T19:49:24.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.160+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd0003520 con 0x7f1ce814be90 2024-03-25T19:49:24.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211035642 0 0) 0x7f1ce80b0d50 con 0x7f1ce80b1730 2024-03-25T19:49:24.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient(hunting): my global_id is 4556 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce80af940 con 0x7f1ce80b1730 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552129669 0 0) 0x7f1cd0003520 con 0x7f1ce814be90 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 >> v1:172.21.15.130:6790/0 conn(0x7f1ce80b1730 legacy=0x7f1ce80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 >> v1:172.21.15.163:6789/0 conn(0x7f1ce80b4d60 legacy=0x7f1ce813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 monclient: found mon.noname-a 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80b1030 con 0x7f1ce814be90 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ce0001a10 con 0x7f1ce814be90 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ced13d640 5 monclient: authenticate success, global_id 4560 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ced13d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ce5d9b640 10 monclient: dump: 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.174+0000 7f1ced13d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:24.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce0001b70 con 0x7f1ce814be90 2024-03-25T19:49:24.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 1 -- 172.21.15.130:0/1061135679 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ce0001f80 con 0x7f1ce814be90 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ce5d9b640 10 monclient: dump: 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:24.184 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:24.185 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ced13d640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:24.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ced13d640 4 set_mon_vals no callback set 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ced13d640 10 monclient: shutdown 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ced13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.184+0000 7f1ced13d640 1 -- 172.21.15.130:0/1061135679 >> v1:172.21.15.130:6789/0 conn(0x7f1ce814be90 legacy=0x7f1ce814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.185+0000 7f1ced13d640 1 -- 172.21.15.130:0/1061135679 shutdown_connections 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.185+0000 7f1ced13d640 1 -- 172.21.15.130:0/1061135679 >> 172.21.15.130:0/1061135679 conn(0x7f1ce80617a0 msgr2=0x7f1ce8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.185+0000 7f1ced13d640 1 -- 172.21.15.130:0/1061135679 shutdown_connections 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.185+0000 7f1ced13d640 1 -- 172.21.15.130:0/1061135679 wait complete. 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 10 monclient: build_initial_monmap 2024-03-25T19:49:24.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 10 monclient: monmap: 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 librados: starting msgr at 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 librados: starting objecter 2024-03-25T19:49:24.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 Processor -- start 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 -- start start 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 librados: setting wanted keys 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 1 librados: calling monclient init 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.186+0000 7f1ced13d640 10 monclient: init 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-c con 0x7f1ce80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-a con 0x7f1ce80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient(hunting): picked mon.noname-b con 0x7f1ce814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8134a00 con 0x7f1ce80b4d60 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce8139050 con 0x7f1ce814be90 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ce813a330 con 0x7f1ce80b1730 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:24.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.187+0000 7f1ced13d640 10 monclient(hunting): authenticate will time out at 5440.333029s 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.201+0000 7f1ce6d9d640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f1ce80b1730 0x7f1ce813ad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:37798/0 (socket says 172.21.15.130:37798) 2024-03-25T19:49:24.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.201+0000 7f1ce6d9d640 1 -- 172.21.15.130:0/2388495124 learned_addr learned my addr 172.21.15.130:0/2388495124 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:24.204 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.205+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce80b1730 2024-03-25T19:49:24.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.205+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce814be90 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3748054257 0 0) 0x7f1ce813a330 con 0x7f1ce80b1730 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 monclient(hunting): my global_id is 4559 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cd4004eb0 con 0x7f1ce80b1730 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 client.?.objecter ms_handle_connect 0x7f1ce80b4d60 2024-03-25T19:49:24.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.215+0000 7f1ccf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:24.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525074424 0 0) 0x7f1ce8139050 con 0x7f1ce814be90 2024-03-25T19:49:24.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 10 monclient(hunting): my global_id is 4561 2024-03-25T19:49:24.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.219+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce813a330 con 0x7f1ce814be90 2024-03-25T19:49:24.235 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2588350980 0 0) 0x7f1ce813a330 con 0x7f1ce814be90 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ce8139050 con 0x7f1ce814be90 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ce0001e80 con 0x7f1ce814be90 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:24.236 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:24.237 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:24.237 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.236+0000 7f1ccf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926794803 0 0) 0x7f1ce8134a00 con 0x7f1ce80b4d60 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 10 monclient(hunting): my global_id is 4563 2024-03-25T19:49:24.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.240+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ce813a330 con 0x7f1ce80b4d60 2024-03-25T19:49:24.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3921832162 0 0) 0x7f1ce8139050 con 0x7f1ce814be90 2024-03-25T19:49:24.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 >> v1:172.21.15.130:6790/0 conn(0x7f1ce80b1730 legacy=0x7f1ce813ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 >> v1:172.21.15.130:6789/0 conn(0x7f1ce80b4d60 legacy=0x7f1ce8133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 1 monclient: found mon.b 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce813a610 con 0x7f1ce814be90 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 5 monclient: authenticate success, global_id 4561 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 10 monclient: _renew_subs 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ce813a8f0 con 0x7f1ce814be90 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 10 monclient: _renew_subs 2024-03-25T19:49:24.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ce8143140 con 0x7f1ce814be90 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.253+0000 7f1ced13d640 1 librados: init done 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.254+0000 7f1ced13d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.254+0000 7f1ced13d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.254+0000 7f1ced13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:24.255 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.254+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ce80627b0 con 0x7f1ce814be90 2024-03-25T19:49:24.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ce0002060 con 0x7f1ce814be90 2024-03-25T19:49:24.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f1ce0003dc0 con 0x7f1ce814be90 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ccf7fe640 10 monclient: dump: 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:24.541 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.540+0000 7f1ce559a640 4 set_mon_vals no callback set 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.542+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f1ce0019d90 con 0x7f1ce814be90 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.542+0000 7f1ce6d9d640 10 monclient: get_auth_request con 0x7f1cd406ddc0 auth_method 0 2024-03-25T19:49:24.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.542+0000 7f1ccf7fe640 10 client.4561.objecter ms_handle_connect 0x7f1cd406ddc0 2024-03-25T19:49:24.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f1ce00538b0 con 0x7f1ce814be90 2024-03-25T19:49:24.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 10 client.4561.objecter ms_dispatch 0x7f1ce8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ce004f1f0 con 0x7f1ce814be90 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:24.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.547+0000 7f1ccf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:24.801 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.802+0000 7f1ced13d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.801 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.802+0000 7f1ced13d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.802+0000 7f1ced13d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:24.802 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.802+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ce81391e0 con 0x7f1ce814be90 2024-03-25T19:49:24.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.837+0000 7f1ccf7fe640 1 -- 172.21.15.130:0/2388495124 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f1ce00512f0 con 0x7f1ce814be90 2024-03-25T19:49:24.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.837+0000 7f1ccf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:24.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.837+0000 7f1ccf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:24.838 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:24.838 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 10 librados: watch_flush enter 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 10 librados: watch_flush exit 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 10 client.4561.objecter successfully canceled tick 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 >> v1:172.21.15.163:6800/1217380380 conn(0x7f1cd406ddc0 legacy=0x7f1cd4070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 10 monclient: shutdown 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 >> v1:172.21.15.163:6789/0 conn(0x7f1ce814be90 legacy=0x7f1ce81342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 shutdown_connections 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 >> 172.21.15.130:0/2388495124 conn(0x7f1ce80617a0 msgr2=0x7f1ce80b6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 shutdown_connections 2024-03-25T19:49:24.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 -- 172.21.15.130:0/2388495124 wait complete. 2024-03-25T19:49:24.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:24.840+0000 7f1ced13d640 1 librados: shutdown 2024-03-25T19:49:24.851 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:30.009 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:14.942 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.056 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:14.989 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.086 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:15.019 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.113 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:15.045 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.117 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:15.050 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.123 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:15.024 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.131 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:15.032 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.164 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:15.097 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:30.167 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:15.068 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:31.658 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:31.775 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.776+0000 7f40b94d3640 10 monclient: get_monmap_and_config 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.776+0000 7f40b94d3640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.776+0000 7f40b94d3640 10 monclient: monmap: 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:31.776998+0000 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:31.776998+0000 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:31.776 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.777+0000 7f40b94d3640 1 Processor -- start 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.777+0000 7f40b94d3640 1 -- start start 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.777+0000 7f40b94d3640 10 monclient: init 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:31.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-c con 0x7f40b40a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-b con 0x7f40b413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-a con 0x7f40b41430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b4142520 con 0x7f40b41430c0 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40b28a0 con 0x7f40b40a98f0 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40b2a30 con 0x7f40b413f710 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.778+0000 7f40b94d3640 10 monclient(hunting): authenticate will time out at 5447.924141s 2024-03-25T19:49:31.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.786+0000 7f40b37fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f40b41430c0 0x7f40b414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:53150/0 (socket says 172.21.15.130:53150) 2024-03-25T19:49:31.785 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.786+0000 7f40b37fe640 1 -- 172.21.15.130:0/703536153 learned_addr learned my addr 172.21.15.130:0/703536153 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:31.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515186646 0 0) 0x7f40b4142520 con 0x7f40b41430c0 2024-03-25T19:49:31.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 10 monclient(hunting): my global_id is 4566 2024-03-25T19:49:31.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.806+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c003520 con 0x7f40b41430c0 2024-03-25T19:49:31.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052681964 0 0) 0x7f40b40b2a30 con 0x7f40b413f710 2024-03-25T19:49:31.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 10 monclient(hunting): my global_id is 4562 2024-03-25T19:49:31.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.808+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b4142520 con 0x7f40b413f710 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981335916 0 0) 0x7f40b40b28a0 con 0x7f40b40a98f0 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 10 monclient(hunting): my global_id is 4564 2024-03-25T19:49:31.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.811+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b40b2a30 con 0x7f40b40a98f0 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3161624249 0 0) 0x7f40b40b2a30 con 0x7f40b40a98f0 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 >> v1:172.21.15.130:6790/0 conn(0x7f40b413f710 legacy=0x7f40b41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 >> v1:172.21.15.130:6789/0 conn(0x7f40b41430c0 legacy=0x7f40b414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 monclient: found mon.noname-c 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40b44c0 con 0x7f40b40a98f0 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f40a40014e0 con 0x7f40b40a98f0 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b94d3640 5 monclient: authenticate success, global_id 4564 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b94d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b1ffb640 10 monclient: dump: 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.827 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:31.828 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:31.828 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:31.828 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:31.828 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:31.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.826+0000 7f40b94d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:31.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a4001640 con 0x7f40b40a98f0 2024-03-25T19:49:31.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:31.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 1 -- 172.21.15.130:0/703536153 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f40a4001a50 con 0x7f40b40a98f0 2024-03-25T19:49:31.839 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b94d3640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b94d3640 4 set_mon_vals no callback set 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b1ffb640 10 monclient: dump: 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:31.840 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.838+0000 7f40b94d3640 10 monclient: shutdown 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 1 -- 172.21.15.130:0/703536153 >> v1:172.21.15.163:6789/0 conn(0x7f40b40a98f0 legacy=0x7f40b40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 1 -- 172.21.15.130:0/703536153 shutdown_connections 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 1 -- 172.21.15.130:0/703536153 >> 172.21.15.130:0/703536153 conn(0x7f40b40b7ea0 msgr2=0x7f40b40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 1 -- 172.21.15.130:0/703536153 shutdown_connections 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 1 -- 172.21.15.130:0/703536153 wait complete. 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.839+0000 7f40b94d3640 10 monclient: build_initial_monmap 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 10 monclient: monmap: 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:31.841 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 librados: starting msgr at 2024-03-25T19:49:31.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 librados: starting objecter 2024-03-25T19:49:31.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 Processor -- start 2024-03-25T19:49:31.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 -- start start 2024-03-25T19:49:31.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 librados: setting wanted keys 2024-03-25T19:49:31.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 1 librados: calling monclient init 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.840+0000 7f40b94d3640 10 monclient: init 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-c con 0x7f40b40a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-b con 0x7f40b413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient(hunting): picked mon.noname-a con 0x7f40b41430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b41dac80 con 0x7f40b41430c0 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b40abe70 con 0x7f40b413f710 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b41da880 con 0x7f40b40a98f0 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:31.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.841+0000 7f40b94d3640 10 monclient(hunting): authenticate will time out at 5447.986984s 2024-03-25T19:49:31.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.859+0000 7f40b27fc640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f40b413f710 0x7f40b41383f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:53876/0 (socket says 172.21.15.130:53876) 2024-03-25T19:49:31.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.859+0000 7f40b27fc640 1 -- 172.21.15.130:0/2259600652 learned_addr learned my addr 172.21.15.130:0/2259600652 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:31.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.860+0000 7f409b7fe640 10 client.?.objecter ms_handle_connect 0x7f40b413f710 2024-03-25T19:49:31.859 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.860+0000 7f409b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.862 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.863+0000 7f409b7fe640 10 client.?.objecter ms_handle_connect 0x7f40b41430c0 2024-03-25T19:49:31.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.863+0000 7f409b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.865+0000 7f409b7fe640 10 client.?.objecter ms_handle_connect 0x7f40b40a98f0 2024-03-25T19:49:31.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.865+0000 7f409b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559410451 0 0) 0x7f40b41da880 con 0x7f40b40a98f0 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 10 monclient(hunting): my global_id is 4565 2024-03-25T19:49:31.875 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.875+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40a0004eb0 con 0x7f40b40a98f0 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847873911 0 0) 0x7f40b40abe70 con 0x7f40b413f710 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T19:49:31.877 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.877+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b41da880 con 0x7f40b413f710 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583896070 0 0) 0x7f40b41dac80 con 0x7f40b41430c0 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T19:49:31.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.883+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40b40abe70 con 0x7f40b41430c0 2024-03-25T19:49:31.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1283685141 0 0) 0x7f40b41da880 con 0x7f40b413f710 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f40b41dac80 con 0x7f40b413f710 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f40a80019f0 con 0x7f40b413f710 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 10 monclient(hunting): dump: 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:31.893 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:31.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.893+0000 7f409b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 317906080 0 0) 0x7f40b41dac80 con 0x7f40b413f710 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 >> v1:172.21.15.130:6790/0 conn(0x7f40b40a98f0 legacy=0x7f40b41db110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 >> v1:172.21.15.130:6789/0 conn(0x7f40b41430c0 legacy=0x7f40b41e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 1 monclient: found mon.b 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:31.912 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b4137b80 con 0x7f40b413f710 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 10 monclient: _finish_auth 0 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f409b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 5 monclient: authenticate success, global_id 4567 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 10 monclient: _renew_subs 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b4138b00 con 0x7f40b413f710 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 10 client.4567.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 10 monclient: _renew_subs 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b41e34b0 con 0x7f40b413f710 2024-03-25T19:49:31.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.912+0000 7f40b94d3640 1 librados: init done 2024-03-25T19:49:31.914 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.913+0000 7f40b94d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.914 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.913+0000 7f40b94d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:31.914 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.913+0000 7f40b94d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:31.914 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:31.913+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b41daa10 con 0x7f40b413f710 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f40a80035f0 con 0x7f40b413f710 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f40a8003a00 con 0x7f40b413f710 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f409b7fe640 10 monclient: dump: 2024-03-25T19:49:32.205 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:32.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.205+0000 7f40b17fa640 4 set_mon_vals no callback set 2024-03-25T19:49:32.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.206+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f40a8003c20 con 0x7f40b413f710 2024-03-25T19:49:32.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.207+0000 7f40b2ffd640 10 monclient: get_auth_request con 0x7f40a006de20 auth_method 0 2024-03-25T19:49:32.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.207+0000 7f409b7fe640 10 client.4567.objecter ms_handle_connect 0x7f40a006de20 2024-03-25T19:49:32.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.212+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f40a8054830 con 0x7f40b413f710 2024-03-25T19:49:32.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.212+0000 7f409b7fe640 10 client.4567.objecter ms_dispatch 0x7f40b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:32.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.212+0000 7f409b7fe640 3 client.4567.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:32.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.212+0000 7f409b7fe640 3 client.4567.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:32.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.212+0000 7f409b7fe640 20 client.4567.objecter dump_active .. 0 homeless 2024-03-25T19:49:32.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.213+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f40a8050340 con 0x7f40b413f710 2024-03-25T19:49:32.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.213+0000 7f409b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:32.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.213+0000 7f409b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:32.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.458+0000 7f40b94d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.458+0000 7f40b94d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.458+0000 7f40b94d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:32.458 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.458+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40b41dae30 con 0x7f40b413f710 2024-03-25T19:49:32.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.566+0000 7f409b7fe640 1 -- 172.21.15.130:0/2259600652 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f40b41dae30 con 0x7f40b413f710 2024-03-25T19:49:32.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.566+0000 7f409b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:32.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.566+0000 7f409b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:32.566 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:32.566 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.568+0000 7f40b94d3640 10 librados: watch_flush enter 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.568+0000 7f40b94d3640 10 librados: watch_flush exit 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.568+0000 7f40b94d3640 20 client.4567.objecter shutdown clearing up homeless session... 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.568+0000 7f40b94d3640 10 client.4567.objecter successfully canceled tick 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 >> v1:172.21.15.163:6800/1217380380 conn(0x7f40a006de20 legacy=0x7f40a00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 10 monclient: shutdown 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 >> v1:172.21.15.163:6789/0 conn(0x7f40b413f710 legacy=0x7f40b41383f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 shutdown_connections 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 >> 172.21.15.130:0/2259600652 conn(0x7f40b40b7ea0 msgr2=0x7f40b4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 shutdown_connections 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 -- 172.21.15.130:0/2259600652 wait complete. 2024-03-25T19:49:32.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:32.569+0000 7f40b94d3640 1 librados: shutdown 2024-03-25T19:49:32.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:39.388 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:39.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.502+0000 7fb82f310640 10 monclient: get_monmap_and_config 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.502+0000 7fb82f310640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.502+0000 7fb82f310640 10 monclient: monmap: 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:39.503732+0000 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:39.503732+0000 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:39.503 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.503+0000 7fb82f310640 1 Processor -- start 2024-03-25T19:49:39.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.504+0000 7fb82f310640 1 -- start start 2024-03-25T19:49:39.504 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.504+0000 7fb82f310640 10 monclient: init 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-a con 0x7fb8280b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-c con 0x7fb8280b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-b con 0x7fb82814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:39.505 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280af940 con 0x7fb8280b4d60 2024-03-25T19:49:39.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280b0a70 con 0x7fb8280b1730 2024-03-25T19:49:39.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280b0d50 con 0x7fb82814be90 2024-03-25T19:49:39.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.505+0000 7fb82f310640 10 monclient(hunting): authenticate will time out at 5455.650985s 2024-03-25T19:49:39.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.509+0000 7fb82d0ac640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fb8280b4d60 0x7fb82813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:36048/0 (socket says 172.21.15.130:36048) 2024-03-25T19:49:39.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.509+0000 7fb82d0ac640 1 -- 172.21.15.130:0/149070807 learned_addr learned my addr 172.21.15.130:0/149070807 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.521+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751920093 0 0) 0x7fb8280b0d50 con 0x7fb82814be90 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): my global_id is 4568 2024-03-25T19:49:39.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb814003520 con 0x7fb82814be90 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592859612 0 0) 0x7fb8280af940 con 0x7fb8280b4d60 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 10 monclient(hunting): my global_id is 4572 2024-03-25T19:49:39.522 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.522+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8280b0d50 con 0x7fb8280b4d60 2024-03-25T19:49:39.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224706878 0 0) 0x7fb814003520 con 0x7fb82814be90 2024-03-25T19:49:39.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 >> v1:172.21.15.163:6789/0 conn(0x7fb8280b1730 legacy=0x7fb8280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:49:39.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 >> v1:172.21.15.130:6789/0 conn(0x7fb8280b4d60 legacy=0x7fb82813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 monclient: found mon.noname-b 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8280b1030 con 0x7fb82814be90 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb820001a10 con 0x7fb82814be90 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb82f310640 5 monclient: authenticate success, global_id 4568 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb82f310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:39.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb81ffff640 10 monclient: dump: 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.529+0000 7fb82f310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb820001b70 con 0x7fb82814be90 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 1 -- 172.21.15.130:0/149070807 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb820001f80 con 0x7fb82814be90 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb82f310640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb82f310640 4 set_mon_vals no callback set 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.533+0000 7fb81ffff640 10 monclient: dump: 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:39.533 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:39.534 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:39.534 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:39.534 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 10 monclient: shutdown 2024-03-25T19:49:39.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 1 -- 172.21.15.130:0/149070807 >> v1:172.21.15.130:6790/0 conn(0x7fb82814be90 legacy=0x7fb82814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 1 -- 172.21.15.130:0/149070807 shutdown_connections 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 1 -- 172.21.15.130:0/149070807 >> 172.21.15.130:0/149070807 conn(0x7fb8280617a0 msgr2=0x7fb828061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 1 -- 172.21.15.130:0/149070807 shutdown_connections 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.534+0000 7fb82f310640 1 -- 172.21.15.130:0/149070807 wait complete. 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 10 monclient: build_initial_monmap 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 10 monclient: monmap: 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:39.536 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 librados: starting msgr at 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 librados: starting objecter 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 Processor -- start 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 -- start start 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 librados: setting wanted keys 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 1 librados: calling monclient init 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.535+0000 7fb82f310640 10 monclient: init 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-a con 0x7fb8280b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-c con 0x7fb8280b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient(hunting): picked mon.noname-b con 0x7fb82814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8280aa7b0 con 0x7fb8280b1730 2024-03-25T19:49:39.537 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82813b740 con 0x7fb82814be90 2024-03-25T19:49:39.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb82813b8d0 con 0x7fb8280b4d60 2024-03-25T19:49:39.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:39.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.536+0000 7fb82f310640 10 monclient(hunting): authenticate will time out at 5455.681805s 2024-03-25T19:49:39.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.542+0000 7fb82d8ad640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb82814be90 0x7fb8281daea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:43558/0 (socket says 172.21.15.130:43558) 2024-03-25T19:49:39.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.543+0000 7fb82d8ad640 1 -- 172.21.15.130:0/3525949048 learned_addr learned my addr 172.21.15.130:0/3525949048 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:39.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.555+0000 7fb81dffb640 10 client.?.objecter ms_handle_connect 0x7fb8280b1730 2024-03-25T19:49:39.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.555+0000 7fb81dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.555+0000 7fb81dffb640 10 client.?.objecter ms_handle_connect 0x7fb82814be90 2024-03-25T19:49:39.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.555+0000 7fb81dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.558+0000 7fb81dffb640 10 client.?.objecter ms_handle_connect 0x7fb8280b4d60 2024-03-25T19:49:39.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.558+0000 7fb81dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:39.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753735064 0 0) 0x7fb82813b740 con 0x7fb82814be90 2024-03-25T19:49:39.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 10 monclient(hunting): my global_id is 4570 2024-03-25T19:49:39.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.562+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c004eb0 con 0x7fb82814be90 2024-03-25T19:49:39.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192097888 0 0) 0x7fb82813b8d0 con 0x7fb8280b4d60 2024-03-25T19:49:39.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 10 monclient(hunting): my global_id is 4571 2024-03-25T19:49:39.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.569+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb82813b740 con 0x7fb8280b4d60 2024-03-25T19:49:39.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054515773 0 0) 0x7fb80c004eb0 con 0x7fb82814be90 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb82813b8d0 con 0x7fb82814be90 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb820001f40 con 0x7fb82814be90 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 10 monclient(hunting): dump: 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.571+0000 7fb81dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 645519986 0 0) 0x7fb8280aa7b0 con 0x7fb8280b1730 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 10 monclient(hunting): _negotiate 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 10 monclient(hunting): my global_id is 4575 2024-03-25T19:49:39.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.572+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb80c004eb0 con 0x7fb8280b1730 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2701246165 0 0) 0x7fb82813b8d0 con 0x7fb82814be90 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 >> v1:172.21.15.130:6790/0 conn(0x7fb8280b4d60 legacy=0x7fb828137f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 >> v1:172.21.15.130:6789/0 conn(0x7fb8280b1730 legacy=0x7fb8280a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 10 monclient: _finish_hunting 0 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 1 monclient: found mon.b 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb828138d60 con 0x7fb82814be90 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 10 monclient: _finish_auth 0 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 10 monclient: _check_auth_tickets 2024-03-25T19:49:39.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb81dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 5 monclient: authenticate success, global_id 4570 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 10 monclient: _renew_subs 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb828138fb0 con 0x7fb82814be90 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 10 monclient: _renew_subs 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb828138860 con 0x7fb82814be90 2024-03-25T19:49:39.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.584+0000 7fb82f310640 1 librados: init done 2024-03-25T19:49:39.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.585+0000 7fb82f310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.585+0000 7fb82f310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.585+0000 7fb82f310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:39.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.585+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8280627b0 con 0x7fb82814be90 2024-03-25T19:49:39.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb820003c20 con 0x7fb82814be90 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb820004030 con 0x7fb82814be90 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81dffb640 10 monclient: dump: 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:39.886 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:39.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.886+0000 7fb81f7fe640 4 set_mon_vals no callback set 2024-03-25T19:49:39.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.887+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fb820004250 con 0x7fb82814be90 2024-03-25T19:49:39.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.887+0000 7fb82d0ac640 10 monclient: get_auth_request con 0x7fb80c06dc10 auth_method 0 2024-03-25T19:49:39.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.888+0000 7fb81dffb640 10 client.4570.objecter ms_handle_connect 0x7fb80c06dc10 2024-03-25T19:49:39.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.890+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fb8200020a0 con 0x7fb82814be90 2024-03-25T19:49:39.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.890+0000 7fb81dffb640 10 client.4570.objecter ms_dispatch 0x7fb828000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:39.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.890+0000 7fb81dffb640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:39.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.890+0000 7fb81dffb640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:39.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.890+0000 7fb81dffb640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T19:49:39.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.892+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb8200505f0 con 0x7fb82814be90 2024-03-25T19:49:39.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.892+0000 7fb81dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:39.892 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:39.892+0000 7fb81dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:40.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.148+0000 7fb82f310640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.148+0000 7fb82f310640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.148+0000 7fb82f310640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:40.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.148+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb82814bc20 con 0x7fb82814be90 2024-03-25T19:49:40.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.174+0000 7fb81dffb640 1 -- 172.21.15.130:0/3525949048 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fb820052520 con 0x7fb82814be90 2024-03-25T19:49:40.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.174+0000 7fb81dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:40.174 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.174+0000 7fb81dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:40.175 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:40.175 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.176+0000 7fb82f310640 10 librados: watch_flush enter 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.176+0000 7fb82f310640 10 librados: watch_flush exit 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.176+0000 7fb82f310640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.176+0000 7fb82f310640 10 client.4570.objecter successfully canceled tick 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 >> v1:172.21.15.163:6800/1217380380 conn(0x7fb80c06dc10 legacy=0x7fb80c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 10 monclient: shutdown 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 >> v1:172.21.15.163:6789/0 conn(0x7fb82814be90 legacy=0x7fb8281daea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 shutdown_connections 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 >> 172.21.15.130:0/3525949048 conn(0x7fb8280617a0 msgr2=0x7fb8280b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 shutdown_connections 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 -- 172.21.15.130:0/3525949048 wait complete. 2024-03-25T19:49:40.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:40.177+0000 7fb82f310640 1 librados: shutdown 2024-03-25T19:49:40.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:46.995 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:47.110 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 10 monclient: get_monmap_and_config 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 10 monclient: build_initial_monmap 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 10 monclient: monmap: 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:47.111921+0000 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:47.111921+0000 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:47.111 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.112 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 1 Processor -- start 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 1 -- start start 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.111+0000 7f20999c4640 10 monclient: init 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-c con 0x7f20940a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-a con 0x7f209413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-b con 0x7f20941430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094142520 con 0x7f209413f710 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20940b28a0 con 0x7f20940a98f0 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20940b2a30 con 0x7f20941430c0 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:47.113 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.112+0000 7f20999c4640 10 monclient(hunting): authenticate will time out at 5463.258154s 2024-03-25T19:49:47.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.116+0000 7f2092ffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f20940a98f0 0x7f20940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:33448/0 (socket says 172.21.15.130:33448) 2024-03-25T19:49:47.115 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.116+0000 7f2092ffd640 1 -- 172.21.15.130:0/1702505786 learned_addr learned my addr 172.21.15.130:0/1702505786 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168820156 0 0) 0x7f20940b28a0 con 0x7f20940a98f0 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T19:49:47.124 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.124+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2080003520 con 0x7f20940a98f0 2024-03-25T19:49:47.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870326253 0 0) 0x7f2094142520 con 0x7f209413f710 2024-03-25T19:49:47.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): my global_id is 4578 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20940b28a0 con 0x7f209413f710 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078028488 0 0) 0x7f20940b2a30 con 0x7f20941430c0 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 10 monclient(hunting): my global_id is 4574 2024-03-25T19:49:47.134 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.133+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2094142520 con 0x7f20941430c0 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3783887237 0 0) 0x7f2080003520 con 0x7f20940a98f0 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 >> v1:172.21.15.130:6790/0 conn(0x7f20941430c0 legacy=0x7f209414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 >> v1:172.21.15.130:6789/0 conn(0x7f209413f710 legacy=0x7f20941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: _finish_hunting 0 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 monclient: found mon.noname-c 2024-03-25T19:49:47.143 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20940b44c0 con 0x7f20940a98f0 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: _finish_auth 0 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: _check_auth_tickets 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2084001690 con 0x7f20940a98f0 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20999c4640 5 monclient: authenticate success, global_id 4573 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20999c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20927fc640 10 monclient: dump: 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.144 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.143+0000 7f20999c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20840017f0 con 0x7f20940a98f0 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 1 -- 172.21.15.130:0/1702505786 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2084001c00 con 0x7f20940a98f0 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20999c4640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20999c4640 4 set_mon_vals no callback set 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:47.154 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.154+0000 7f20927fc640 10 monclient: dump: 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:47.155 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 10 monclient: shutdown 2024-03-25T19:49:47.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 1 -- 172.21.15.130:0/1702505786 >> v1:172.21.15.163:6789/0 conn(0x7f20940a98f0 legacy=0x7f20940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 1 -- 172.21.15.130:0/1702505786 shutdown_connections 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 1 -- 172.21.15.130:0/1702505786 >> 172.21.15.130:0/1702505786 conn(0x7f20940b7ea0 msgr2=0x7f20940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 1 -- 172.21.15.130:0/1702505786 shutdown_connections 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.155+0000 7f20999c4640 1 -- 172.21.15.130:0/1702505786 wait complete. 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.156+0000 7f20999c4640 10 monclient: build_initial_monmap 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.156+0000 7f20999c4640 10 monclient: monmap: 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:47.157 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.156+0000 7f20999c4640 1 librados: starting msgr at 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.156+0000 7f20999c4640 1 librados: starting objecter 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.157+0000 7f20999c4640 1 Processor -- start 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.157+0000 7f20999c4640 1 -- start start 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.157+0000 7f20999c4640 1 librados: setting wanted keys 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.157+0000 7f20999c4640 1 librados: calling monclient init 2024-03-25T19:49:47.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.157+0000 7f20999c4640 10 monclient: init 2024-03-25T19:49:47.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:47.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:49:47.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-b con 0x7f20940a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:47.159 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-c con 0x7f209413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient(hunting): picked mon.noname-a con 0x7f20941430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094138b40 con 0x7f20941430c0 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094138e20 con 0x7f20940a98f0 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2094139100 con 0x7f209413f710 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:47.160 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.158+0000 7f20999c4640 10 monclient(hunting): authenticate will time out at 5463.304052s 2024-03-25T19:49:47.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.167+0000 7f2092ffd640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f20940a98f0 0x7f2094137380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:33490/0 (socket says 172.21.15.130:33490) 2024-03-25T19:49:47.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.168+0000 7f2092ffd640 1 -- 172.21.15.130:0/3083370682 learned_addr learned my addr 172.21.15.130:0/3083370682 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:47.167 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.167+0000 7f20937fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f20941430c0 0x7f20941e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:35700/0 (socket says 172.21.15.130:35700) 2024-03-25T19:49:47.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.168+0000 7f2090ff9640 10 client.?.objecter ms_handle_connect 0x7f20940a98f0 2024-03-25T19:49:47.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.168+0000 7f2090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.169+0000 7f2090ff9640 10 client.?.objecter ms_handle_connect 0x7f20941430c0 2024-03-25T19:49:47.168 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.169+0000 7f2090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2493444640 0 0) 0x7f2094138e20 con 0x7f20940a98f0 2024-03-25T19:49:47.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 monclient(hunting): my global_id is 4576 2024-03-25T19:49:47.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2074004eb0 con 0x7f20940a98f0 2024-03-25T19:49:47.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 client.?.objecter ms_handle_connect 0x7f209413f710 2024-03-25T19:49:47.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.181+0000 7f2090ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:47.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430123331 0 0) 0x7f2094138b40 con 0x7f20941430c0 2024-03-25T19:49:47.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 10 monclient(hunting): my global_id is 4581 2024-03-25T19:49:47.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.194+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2094138e20 con 0x7f20941430c0 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426795760 0 0) 0x7f2094139100 con 0x7f209413f710 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 10 monclient(hunting): my global_id is 4577 2024-03-25T19:49:47.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.206+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2094138b40 con 0x7f209413f710 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99418565 0 0) 0x7f2074004eb0 con 0x7f20940a98f0 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2094139100 con 0x7f20940a98f0 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2084001bc0 con 0x7f20940a98f0 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.207 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.207+0000 7f2090ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 50896964 0 0) 0x7f2094138e20 con 0x7f20941430c0 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2074004eb0 con 0x7f20941430c0 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f208c001b50 con 0x7f20941430c0 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.217+0000 7f2090ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.217 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:47.218 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:47.218 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:47.218 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2669562219 0 0) 0x7f2094139100 con 0x7f20940a98f0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 >> v1:172.21.15.130:6790/0 conn(0x7f209413f710 legacy=0x7f2094137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 >> v1:172.21.15.130:6789/0 conn(0x7f20941430c0 legacy=0x7f20941e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 1 monclient: found mon.b 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20941397a0 con 0x7f20940a98f0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f2090ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f20999c4640 5 monclient: authenticate success, global_id 4576 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f20999c4640 10 monclient: _renew_subs 2024-03-25T19:49:47.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f20999c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.229+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2094136aa0 con 0x7f20940a98f0 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 10 monclient: _renew_subs 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f209413c320 con 0x7f20940a98f0 2024-03-25T19:49:47.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 1 librados: init done 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.230+0000 7f20999c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f20999c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f20999c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20941395b0 con 0x7f20940a98f0 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20840011f0 con 0x7f20940a98f0 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f2090ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:47.232 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.231+0000 7f2091ffb640 4 set_mon_vals no callback set 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.524+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2084004c30 con 0x7f20940a98f0 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.524+0000 7f2090ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.524+0000 7f2090ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.524+0000 7f2090ff9640 10 monclient: dump: 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:47.524 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.524+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f208401ac80 con 0x7f20940a98f0 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.525+0000 7f208a7fc640 10 monclient: get_auth_request con 0x7f207406ded0 auth_method 0 2024-03-25T19:49:47.525 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.525+0000 7f2090ff9640 10 client.4576.objecter ms_handle_connect 0x7f207406ded0 2024-03-25T19:49:47.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.527+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f20840549f0 con 0x7f20940a98f0 2024-03-25T19:49:47.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.527+0000 7f2090ff9640 10 client.4576.objecter ms_dispatch 0x7f2094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:47.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.527+0000 7f2090ff9640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:47.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.527+0000 7f2090ff9640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:47.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.527+0000 7f2090ff9640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T19:49:47.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.531+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2084050330 con 0x7f20940a98f0 2024-03-25T19:49:47.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.531+0000 7f2090ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:47.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.531+0000 7f2090ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.765+0000 7f20999c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.765+0000 7f20999c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.765+0000 7f20999c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:47.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.765+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20940627b0 con 0x7f20940a98f0 2024-03-25T19:49:47.803 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.804+0000 7f2090ff9640 1 -- 172.21.15.130:0/3083370682 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f2084052430 con 0x7f20940a98f0 2024-03-25T19:49:47.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.804+0000 7f2090ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:47.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.804+0000 7f2090ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:47.804 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:47.804 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 10 librados: watch_flush enter 2024-03-25T19:49:47.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 10 librados: watch_flush exit 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 10 client.4576.objecter successfully canceled tick 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 >> v1:172.21.15.163:6800/1217380380 conn(0x7f207406ded0 legacy=0x7f2074070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 10 monclient: shutdown 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.806+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 >> v1:172.21.15.163:6789/0 conn(0x7f20940a98f0 legacy=0x7f2094137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.807+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 shutdown_connections 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.807+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 >> 172.21.15.130:0/3083370682 conn(0x7f20940b7ea0 msgr2=0x7f209414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.807+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 shutdown_connections 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.807+0000 7f20999c4640 1 -- 172.21.15.130:0/3083370682 wait complete. 2024-03-25T19:49:47.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:47.807+0000 7f20999c4640 1 librados: shutdown 2024-03-25T19:49:47.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:49:50.363 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:35.295 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.410 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:35.343 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.411 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:35.344 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.413 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:35.346 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.442 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:35.374 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.445 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:35.379 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.462 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:35.363 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.476 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:35.377 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:50.502 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:35.403 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:49:54.624 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:49:54.728 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.729+0000 7f8cee956640 10 monclient: get_monmap_and_config 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.729+0000 7f8cee956640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.729+0000 7f8cee956640 10 monclient: monmap: 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:49:54.730208+0000 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:49:54.730208+0000 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:49:54.729 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:54.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 1 Processor -- start 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 1 -- start start 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient: init 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce80b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce80b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80af940 con 0x7f8ce814be90 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80b0a70 con 0x7f8ce80b1730 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce80b0d50 con 0x7f8ce80b4d60 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.731 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.730+0000 7f8cee956640 10 monclient(hunting): authenticate will time out at 5470.876470s 2024-03-25T19:49:54.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.753+0000 7f8ce7fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f8ce80b4d60 0x7f8ce813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:50242/0 (socket says 172.21.15.130:50242) 2024-03-25T19:49:54.752 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.753+0000 7f8ce7fff640 1 -- 172.21.15.130:0/84231342 learned_addr learned my addr 172.21.15.130:0/84231342 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:54.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3604895440 0 0) 0x7f8ce80b0a70 con 0x7f8ce80b1730 2024-03-25T19:49:54.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 10 monclient(hunting): my global_id is 4579 2024-03-25T19:49:54.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.771+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd0003520 con 0x7f8ce80b1730 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1030790088 0 0) 0x7f8ce80b0d50 con 0x7f8ce80b4d60 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 10 monclient(hunting): my global_id is 4580 2024-03-25T19:49:54.776 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.776+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce80b0a70 con 0x7f8ce80b4d60 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2389596866 0 0) 0x7f8ce80af940 con 0x7f8ce814be90 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 10 monclient(hunting): my global_id is 4584 2024-03-25T19:49:54.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.782+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce80b0d50 con 0x7f8ce814be90 2024-03-25T19:49:54.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 201823785 0 0) 0x7f8cd0003520 con 0x7f8ce80b1730 2024-03-25T19:49:54.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 >> v1:172.21.15.130:6790/0 conn(0x7f8ce80b4d60 legacy=0x7f8ce813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 >> v1:172.21.15.130:6789/0 conn(0x7f8ce814be90 legacy=0x7f8ce814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.787 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce80b1030 con 0x7f8ce80b1730 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8cdc001690 con 0x7f8ce80b1730 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8ce6ffd640 10 monclient: dump: 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:54.788 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:54.789 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:54.789 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:54.789 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:54.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8cee956640 5 monclient: authenticate success, global_id 4579 2024-03-25T19:49:54.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.787+0000 7f8cee956640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:49:54.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8cdc0017f0 con 0x7f8ce80b1730 2024-03-25T19:49:54.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:54.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 1 -- 172.21.15.130:0/84231342 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8cdc001c00 con 0x7f8ce80b1730 2024-03-25T19:49:54.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 10 monclient: get_monmap_and_config success 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 4 set_mon_vals no callback set 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8ce6ffd640 10 monclient: dump: 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:54.794 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:54.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 10 monclient: shutdown 2024-03-25T19:49:54.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:54.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 1 -- 172.21.15.130:0/84231342 >> v1:172.21.15.163:6789/0 conn(0x7f8ce80b1730 legacy=0x7f8ce80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 1 -- 172.21.15.130:0/84231342 shutdown_connections 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 1 -- 172.21.15.130:0/84231342 >> 172.21.15.130:0/84231342 conn(0x7f8ce80617a0 msgr2=0x7f8ce8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 1 -- 172.21.15.130:0/84231342 shutdown_connections 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.794+0000 7f8cee956640 1 -- 172.21.15.130:0/84231342 wait complete. 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.795+0000 7f8cee956640 10 monclient: build_initial_monmap 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.795+0000 7f8cee956640 10 monclient: monmap: 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:49:54.797 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 librados: starting msgr at 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 librados: starting objecter 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 Processor -- start 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 -- start start 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 librados: setting wanted keys 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 librados: calling monclient init 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient: init 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient: _reopen_session rank -1 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-a con 0x7f8ce80b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-c con 0x7f8ce80b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient(hunting): picked mon.noname-b con 0x7f8ce814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce8137550 con 0x7f8ce80b1730 2024-03-25T19:49:54.798 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce8137830 con 0x7f8ce814be90 2024-03-25T19:49:54.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce8143260 con 0x7f8ce80b4d60 2024-03-25T19:49:54.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient(hunting): _renew_subs 2024-03-25T19:49:54.799 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.796+0000 7f8cee956640 10 monclient(hunting): authenticate will time out at 5470.942588s 2024-03-25T19:49:54.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.806+0000 7f8cecef3640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f8ce814be90 0x7f8ce8138360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:38220/0 (socket says 172.21.15.130:38220) 2024-03-25T19:49:54.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.806+0000 7f8cecef3640 1 -- 172.21.15.130:0/1572721526 learned_addr learned my addr 172.21.15.130:0/1572721526 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:49:54.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.809+0000 7f8ce4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ce814be90 2024-03-25T19:49:54.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.809+0000 7f8ce4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.814+0000 7f8ce4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ce80b1730 2024-03-25T19:49:54.814 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.814+0000 7f8ce4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217183531 0 0) 0x7f8ce8137830 con 0x7f8ce814be90 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 10 monclient(hunting): my global_id is 4582 2024-03-25T19:49:54.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.818+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd4004eb0 con 0x7f8ce814be90 2024-03-25T19:49:54.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.829+0000 7f8ce4ff9640 10 client.?.objecter ms_handle_connect 0x7f8ce80b4d60 2024-03-25T19:49:54.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.830+0000 7f8ce4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1550619666 0 0) 0x7f8cd4004eb0 con 0x7f8ce814be90 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ce8137830 con 0x7f8ce814be90 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8ce0001c70 con 0x7f8ce814be90 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 10 monclient(hunting): dump: 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:54.833 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.833+0000 7f8ce4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281184638 0 0) 0x7f8ce8137550 con 0x7f8ce80b1730 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:49:54.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:49:54.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:49:54.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 10 monclient(hunting): my global_id is 4587 2024-03-25T19:49:54.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.834+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cd4004eb0 con 0x7f8ce80b1730 2024-03-25T19:49:54.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3585276151 0 0) 0x7f8ce8137830 con 0x7f8ce814be90 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 >> v1:172.21.15.130:6790/0 conn(0x7f8ce80b4d60 legacy=0x7f8ce8137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 >> v1:172.21.15.130:6789/0 conn(0x7f8ce80b1730 legacy=0x7f8ce81db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 1 monclient: found mon.b 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce8138a70 con 0x7f8ce814be90 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8ce4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 5 monclient: authenticate success, global_id 4582 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 10 monclient: _renew_subs 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:54.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce8135c10 con 0x7f8ce814be90 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 10 monclient: _renew_subs 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce8142eb0 con 0x7f8ce814be90 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.842+0000 7f8cee956640 1 librados: init done 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.843+0000 7f8cee956640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.843+0000 7f8cee956640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.843+0000 7f8cee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:54.843 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:54.843+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce80627b0 con 0x7f8ce814be90 2024-03-25T19:49:55.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ce0003990 con 0x7f8ce814be90 2024-03-25T19:49:55.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:49:55.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8ce0003de0 con 0x7f8ce814be90 2024-03-25T19:49:55.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:49:55.131 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce4ff9640 10 monclient: dump: 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:49:55.132 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.131+0000 7f8ce67fc640 4 set_mon_vals no callback set 2024-03-25T19:49:55.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.132+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8ce0004000 con 0x7f8ce814be90 2024-03-25T19:49:55.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.133+0000 7f8ce7fff640 10 monclient: get_auth_request con 0x7f8cd406dd90 auth_method 0 2024-03-25T19:49:55.133 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.133+0000 7f8ce4ff9640 10 client.4582.objecter ms_handle_connect 0x7f8cd406dd90 2024-03-25T19:49:55.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.139+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8ce0054a00 con 0x7f8ce814be90 2024-03-25T19:49:55.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.139+0000 7f8ce4ff9640 10 client.4582.objecter ms_dispatch 0x7f8ce8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:49:55.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.139+0000 7f8ce4ff9640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:49:55.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.139+0000 7f8ce4ff9640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:49:55.139 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.139+0000 7f8ce4ff9640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T19:49:55.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.141+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ce0050510 con 0x7f8ce814be90 2024-03-25T19:49:55.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.141+0000 7f8ce4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:49:55.140 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.141+0000 7f8ce4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:49:55.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.392+0000 7f8cee956640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.392+0000 7f8cee956640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.392+0000 7f8cee956640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:49:55.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.392+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ce8140170 con 0x7f8ce814be90 2024-03-25T19:49:55.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.418+0000 7f8ce4ff9640 1 -- 172.21.15.130:0/1572721526 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8ce0052440 con 0x7f8ce814be90 2024-03-25T19:49:55.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.418+0000 7f8ce4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:49:55.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.418+0000 7f8ce4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:49:55.418 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:49:55.418 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 10 librados: watch_flush enter 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 10 librados: watch_flush exit 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 10 client.4582.objecter successfully canceled tick 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 >> v1:172.21.15.163:6800/1217380380 conn(0x7f8cd406dd90 legacy=0x7f8cd4070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 10 monclient: shutdown 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.420+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 >> v1:172.21.15.163:6789/0 conn(0x7f8ce814be90 legacy=0x7f8ce8138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.421+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 shutdown_connections 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.421+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 >> 172.21.15.130:0/1572721526 conn(0x7f8ce80617a0 msgr2=0x7f8ce80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.421+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 shutdown_connections 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.421+0000 7f8cee956640 1 -- 172.21.15.130:0/1572721526 wait complete. 2024-03-25T19:49:55.421 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:49:55.421+0000 7f8cee956640 1 librados: shutdown 2024-03-25T19:49:55.431 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:02.236 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.348+0000 7f05b2fc1640 10 monclient: get_monmap_and_config 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.348+0000 7f05b2fc1640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.348+0000 7f05b2fc1640 10 monclient: monmap: 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:02.349 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:02.350765+0000 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:02.350765+0000 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:02.350 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.349+0000 7f05b2fc1640 1 Processor -- start 2024-03-25T19:50:02.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.349+0000 7f05b2fc1640 1 -- start start 2024-03-25T19:50:02.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.349+0000 7f05b2fc1640 10 monclient: init 2024-03-25T19:50:02.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f05ac0b10d0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f05ac0b4360 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f05ac0a9470 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0a9ab0 con 0x7f05ac0a9470 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0b2400 con 0x7f05ac0b4360 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0aa2a0 con 0x7f05ac0b10d0 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05b2fc1640 10 monclient(hunting): authenticate will time out at 5478.496971s 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.350+0000 7f05abfff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f05ac0b4360 0x7f05ac0b1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:35750/0 (socket says 172.21.15.130:35750) 2024-03-25T19:50:02.352 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.351+0000 7f05abfff640 1 -- 172.21.15.130:0/2807279763 learned_addr learned my addr 172.21.15.130:0/2807279763 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:02.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326591881 0 0) 0x7f05ac0aa2a0 con 0x7f05ac0b10d0 2024-03-25T19:50:02.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.398 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 10 monclient(hunting): my global_id is 4586 2024-03-25T19:50:02.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.398+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059c003520 con 0x7f05ac0b10d0 2024-03-25T19:50:02.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 167120755 0 0) 0x7f059c003520 con 0x7f05ac0b10d0 2024-03-25T19:50:02.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 >> v1:172.21.15.163:6789/0 conn(0x7f05ac0b4360 legacy=0x7f05ac0b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 >> v1:172.21.15.130:6789/0 conn(0x7f05ac0a9470 legacy=0x7f05ac14dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 monclient: found mon.noname-b 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac0b14b0 con 0x7f05ac0b10d0 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f05a0001690 con 0x7f05ac0b10d0 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05ab7fe640 10 monclient: dump: 2024-03-25T19:50:02.411 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05b2fc1640 5 monclient: authenticate success, global_id 4586 2024-03-25T19:50:02.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.409+0000 7f05b2fc1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05a00017f0 con 0x7f05ac0b10d0 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 1 -- 172.21.15.130:0/2807279763 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f05a0001c00 con 0x7f05ac0b10d0 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05ab7fe640 10 monclient: dump: 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:02.424 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05b2fc1640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:02.425 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.423+0000 7f05b2fc1640 4 set_mon_vals no callback set 2024-03-25T19:50:02.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 10 monclient: shutdown 2024-03-25T19:50:02.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:02.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2807279763 >> v1:172.21.15.130:6790/0 conn(0x7f05ac0b10d0 legacy=0x7f05ac0b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2807279763 shutdown_connections 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2807279763 >> 172.21.15.130:0/2807279763 conn(0x7f05ac0617a0 msgr2=0x7f05ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2807279763 shutdown_connections 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.424+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2807279763 wait complete. 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 10 monclient: build_initial_monmap 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 10 monclient: monmap: 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:02.428 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 librados: starting msgr at 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 librados: starting objecter 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 Processor -- start 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 -- start start 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 librados: setting wanted keys 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 1 librados: calling monclient init 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.425+0000 7f05b2fc1640 10 monclient: init 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-a con 0x7f05ac0a9470 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-c con 0x7f05ac0b10d0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient(hunting): picked mon.noname-b con 0x7f05ac0b4360 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0ab8a0 con 0x7f05ac0a9470 2024-03-25T19:50:02.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac0aba30 con 0x7f05ac0b4360 2024-03-25T19:50:02.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05ac13e820 con 0x7f05ac0b10d0 2024-03-25T19:50:02.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:02.430 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.426+0000 7f05b2fc1640 10 monclient(hunting): authenticate will time out at 5478.573383s 2024-03-25T19:50:02.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.434+0000 7f05b0d5d640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f05ac0a9470 0x7f05ac138df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:45274/0 (socket says 172.21.15.130:45274) 2024-03-25T19:50:02.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.434+0000 7f05b0d5d640 1 -- 172.21.15.130:0/2699726893 learned_addr learned my addr 172.21.15.130:0/2699726893 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:02.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.435+0000 7f05a97fa640 10 client.?.objecter ms_handle_connect 0x7f05ac0a9470 2024-03-25T19:50:02.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.435+0000 7f05a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 client.?.objecter ms_handle_connect 0x7f05ac0b10d0 2024-03-25T19:50:02.445 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1344592456 0 0) 0x7f05ac0ab8a0 con 0x7f05ac0a9470 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 10 monclient(hunting): my global_id is 4593 2024-03-25T19:50:02.446 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.445+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05ac0a9470 2024-03-25T19:50:02.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.455+0000 7f05a97fa640 10 client.?.objecter ms_handle_connect 0x7f05ac0b4360 2024-03-25T19:50:02.455 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.455+0000 7f05a97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3609416667 0 0) 0x7f05ac0aba30 con 0x7f05ac0b4360 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 10 monclient(hunting): my global_id is 4588 2024-03-25T19:50:02.461 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.460+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05ac0ab8a0 con 0x7f05ac0b4360 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2522657980 0 0) 0x7f0590004eb0 con 0x7f05ac0a9470 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05ac0aba30 con 0x7f05ac0a9470 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f05a0001c00 con 0x7f05ac0a9470 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 10 monclient(hunting): dump: 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:02.472 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.473 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.471+0000 7f05a97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875031935 0 0) 0x7f05ac13e820 con 0x7f05ac0b10d0 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 10 monclient(hunting): _negotiate 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T19:50:02.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.474+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05ac0b10d0 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3312738911 0 0) 0x7f05ac0ab8a0 con 0x7f05ac0b4360 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05ac13e820 con 0x7f05ac0b4360 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f05a4001b10 con 0x7f05ac0b4360 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.481+0000 7f05a97fa640 10 monclient(hunting): dump: 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:02.482 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:02.483 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1778541681 0 0) 0x7f05ac13e820 con 0x7f05ac0b4360 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 >> v1:172.21.15.130:6790/0 conn(0x7f05ac0b10d0 legacy=0x7f05ac1dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 >> v1:172.21.15.130:6789/0 conn(0x7f05ac0a9470 legacy=0x7f05ac138df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 10 monclient: _finish_hunting 0 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 1 monclient: found mon.b 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05ac13e9b0 con 0x7f05ac0b4360 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 10 monclient: _finish_auth 0 2024-03-25T19:50:02.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 10 monclient: _check_auth_tickets 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 5 monclient: authenticate success, global_id 4588 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 10 monclient: _renew_subs 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05ac139500 con 0x7f05ac0b4360 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 10 monclient: _renew_subs 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05ac13da80 con 0x7f05ac0b4360 2024-03-25T19:50:02.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.496+0000 7f05b2fc1640 1 librados: init done 2024-03-25T19:50:02.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.498+0000 7f05b2fc1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.498+0000 7f05b2fc1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.498+0000 7f05b2fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:02.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.498+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05ac13eba0 con 0x7f05ac0b4360 2024-03-25T19:50:02.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.717+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05a4002000 con 0x7f05ac0b4360 2024-03-25T19:50:02.718 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.717+0000 7f05a97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:02.718 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.717+0000 7f05aaffd640 4 set_mon_vals no callback set 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.787+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f05a4003b30 con 0x7f05ac0b4360 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.787+0000 7f05a97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.787+0000 7f05a97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.787+0000 7f05a97fa640 10 monclient: dump: 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.788 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:02.789 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.788+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f05a4003d10 con 0x7f05ac0b4360 2024-03-25T19:50:02.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.789+0000 7f05b0d5d640 10 monclient: get_auth_request con 0x7f059006df20 auth_method 0 2024-03-25T19:50:02.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.789+0000 7f05a97fa640 10 client.4588.objecter ms_handle_connect 0x7f059006df20 2024-03-25T19:50:02.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.792+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f05a4053800 con 0x7f05ac0b4360 2024-03-25T19:50:02.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.792+0000 7f05a97fa640 10 client.4588.objecter ms_dispatch 0x7f05ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:02.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.792+0000 7f05a97fa640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:02.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.792+0000 7f05a97fa640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:02.793 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.792+0000 7f05a97fa640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T19:50:02.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.795+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f05a404f310 con 0x7f05ac0b4360 2024-03-25T19:50:02.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.795+0000 7f05a97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:02.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:02.795+0000 7f05a97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:03.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.038+0000 7f05b2fc1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.039+0000 7f05b2fc1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.039+0000 7f05b2fc1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:03.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.039+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f05ac0627b0 con 0x7f05ac0b4360 2024-03-25T19:50:03.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.061+0000 7f05a97fa640 1 -- 172.21.15.130:0/2699726893 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f05a4051240 con 0x7f05ac0b4360 2024-03-25T19:50:03.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.061+0000 7f05a97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:03.062 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.061+0000 7f05a97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:03.062 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:03.062 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 10 librados: watch_flush enter 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 10 librados: watch_flush exit 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 10 client.4588.objecter successfully canceled tick 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 >> v1:172.21.15.163:6800/1217380380 conn(0x7f059006df20 legacy=0x7f05900703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 10 monclient: shutdown 2024-03-25T19:50:03.064 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 >> v1:172.21.15.163:6789/0 conn(0x7f05ac0b4360 legacy=0x7f05ac1db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 shutdown_connections 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 >> 172.21.15.130:0/2699726893 conn(0x7f05ac0617a0 msgr2=0x7f05ac0b2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 shutdown_connections 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 -- 172.21.15.130:0/2699726893 wait complete. 2024-03-25T19:50:03.065 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:03.063+0000 7f05b2fc1640 1 librados: shutdown 2024-03-25T19:50:03.076 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:09.882 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:10.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.080+0000 7f8143dd7640 10 monclient: get_monmap_and_config 2024-03-25T19:50:10.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.080+0000 7f8143dd7640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.080 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.080+0000 7f8143dd7640 10 monclient: monmap: 2024-03-25T19:50:10.080 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:10.080 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:10.081702+0000 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:10.081702+0000 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:10.081 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.081+0000 7f8143dd7640 1 Processor -- start 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.081+0000 7f8143dd7640 1 -- start start 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.081+0000 7f8143dd7640 10 monclient: init 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f813c0b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f813c0b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f813c14be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:10.082 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c0af940 con 0x7f813c0b1730 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c0b0a70 con 0x7f813c14be90 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c0b0d50 con 0x7f813c0b4d60 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.083 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.082+0000 7f8143dd7640 10 monclient(hunting): authenticate will time out at 5486.228111s 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.090+0000 7f8142374640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f813c14be90 0x7f813c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51826/0 (socket says 172.21.15.130:51826) 2024-03-25T19:50:10.089 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.090+0000 7f8142374640 1 -- 172.21.15.130:0/2559007174 learned_addr learned my addr 172.21.15.130:0/2559007174 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1343375085 0 0) 0x7f813c0b0a70 con 0x7f813c14be90 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 10 monclient(hunting): my global_id is 4591 2024-03-25T19:50:10.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.106+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8128003520 con 0x7f813c14be90 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088133532 0 0) 0x7f813c0b0d50 con 0x7f813c0b4d60 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 10 monclient(hunting): my global_id is 4592 2024-03-25T19:50:10.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.166+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f813c0b0a70 con 0x7f813c0b4d60 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2634556523 0 0) 0x7f813c0b0a70 con 0x7f813c0b4d60 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 >> v1:172.21.15.163:6789/0 conn(0x7f813c14be90 legacy=0x7f813c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 >> v1:172.21.15.130:6789/0 conn(0x7f813c0b1730 legacy=0x7f813c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 1 monclient: found mon.noname-b 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.207+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c0b1030 con 0x7f813c0b4d60 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8138001670 con 0x7f813c0b4d60 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:10.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8140b71640 10 monclient: dump: 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8143dd7640 5 monclient: authenticate success, global_id 4592 2024-03-25T19:50:10.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.208+0000 7f8143dd7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:10.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81380017d0 con 0x7f813c0b4d60 2024-03-25T19:50:10.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 1 -- 172.21.15.130:0/2559007174 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8138001ba0 con 0x7f813c0b4d60 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 4 set_mon_vals no callback set 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8140b71640 10 monclient: dump: 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:10.225 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:10.226 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:10.226 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 10 monclient: shutdown 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 1 -- 172.21.15.130:0/2559007174 >> v1:172.21.15.130:6790/0 conn(0x7f813c0b4d60 legacy=0x7f813c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 1 -- 172.21.15.130:0/2559007174 shutdown_connections 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.225+0000 7f8143dd7640 1 -- 172.21.15.130:0/2559007174 >> 172.21.15.130:0/2559007174 conn(0x7f813c0617a0 msgr2=0x7f813c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.226+0000 7f8143dd7640 1 -- 172.21.15.130:0/2559007174 shutdown_connections 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.226+0000 7f8143dd7640 1 -- 172.21.15.130:0/2559007174 wait complete. 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 10 monclient: build_initial_monmap 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 10 monclient: monmap: 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:10.228 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 librados: starting msgr at 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 librados: starting objecter 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 Processor -- start 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 -- start start 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 librados: setting wanted keys 2024-03-25T19:50:10.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 1 librados: calling monclient init 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.227+0000 7f8143dd7640 10 monclient: init 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-a con 0x7f813c0b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-c con 0x7f813c0b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient(hunting): picked mon.noname-b con 0x7f813c14be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c141c80 con 0x7f813c0b1730 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c137d20 con 0x7f813c14be90 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f813c140a20 con 0x7f813c0b4d60 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:10.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.228+0000 7f8143dd7640 10 monclient(hunting): authenticate will time out at 5486.374171s 2024-03-25T19:50:10.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.233+0000 7f8142374640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f813c14be90 0x7f813c1e2f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51882/0 (socket says 172.21.15.130:51882) 2024-03-25T19:50:10.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.233+0000 7f8142374640 1 -- 172.21.15.130:0/4172250406 learned_addr learned my addr 172.21.15.130:0/4172250406 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:10.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.242+0000 7f81327fc640 10 client.?.objecter ms_handle_connect 0x7f813c14be90 2024-03-25T19:50:10.242 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.242+0000 7f81327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.243 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.244+0000 7f81327fc640 10 client.?.objecter ms_handle_connect 0x7f813c0b4d60 2024-03-25T19:50:10.243 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.244+0000 7f81327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.253 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3306979132 0 0) 0x7f813c137d20 con 0x7f813c14be90 2024-03-25T19:50:10.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 10 monclient(hunting): my global_id is 4594 2024-03-25T19:50:10.254 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.254+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8120004eb0 con 0x7f813c14be90 2024-03-25T19:50:10.259 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.259+0000 7f81327fc640 10 client.?.objecter ms_handle_connect 0x7f813c0b1730 2024-03-25T19:50:10.259 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.259+0000 7f81327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459701316 0 0) 0x7f813c140a20 con 0x7f813c0b4d60 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T19:50:10.262 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.262+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f813c137d20 con 0x7f813c0b4d60 2024-03-25T19:50:10.263 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1358515818 0 0) 0x7f8120004eb0 con 0x7f813c14be90 2024-03-25T19:50:10.263 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f813c140a20 con 0x7f813c14be90 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8134001b30 con 0x7f813c14be90 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 10 monclient(hunting): dump: 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.264 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.264+0000 7f81327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4170152780 0 0) 0x7f813c141c80 con 0x7f813c0b1730 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 10 monclient(hunting): my global_id is 4599 2024-03-25T19:50:10.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.271+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8120004eb0 con 0x7f813c0b1730 2024-03-25T19:50:10.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 715638390 0 0) 0x7f813c140a20 con 0x7f813c14be90 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 >> v1:172.21.15.130:6790/0 conn(0x7f813c0b4d60 legacy=0x7f813c1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 >> v1:172.21.15.130:6789/0 conn(0x7f813c0b1730 legacy=0x7f813c131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 1 monclient: found mon.b 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f813c132620 con 0x7f813c14be90 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f81327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 5 monclient: authenticate success, global_id 4594 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 10 monclient: _renew_subs 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f813c132810 con 0x7f813c14be90 2024-03-25T19:50:10.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 10 monclient: _renew_subs 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f813c1db4d0 con 0x7f813c14be90 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.272+0000 7f8143dd7640 1 librados: init done 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.273+0000 7f8143dd7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.274 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.273+0000 7f8143dd7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.273+0000 7f8143dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:10.275 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.273+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f813c0b02f0 con 0x7f813c14be90 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8134001fe0 con 0x7f813c14be90 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8134004d00 con 0x7f813c14be90 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:10.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f81327fc640 10 monclient: dump: 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:10.295 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.294+0000 7f8133fff640 4 set_mon_vals no callback set 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.296+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8134004ee0 con 0x7f813c14be90 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.296+0000 7f8141b73640 10 monclient: get_auth_request con 0x7f812006de00 auth_method 0 2024-03-25T19:50:10.296 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.296+0000 7f81327fc640 10 client.4594.objecter ms_handle_connect 0x7f812006de00 2024-03-25T19:50:10.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.298+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8134054a50 con 0x7f813c14be90 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.298+0000 7f81327fc640 10 client.4594.objecter ms_dispatch 0x7f813c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.298+0000 7f81327fc640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.298+0000 7f81327fc640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:10.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.298+0000 7f81327fc640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T19:50:10.557 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.558+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8134050390 con 0x7f813c14be90 2024-03-25T19:50:10.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.558+0000 7f81327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:10.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.558+0000 7f81327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:10.637 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:55.569 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.649 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:55.582 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.678 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:08:55.610 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.686 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:55.587 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.694 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:55.595 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.698 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:55.631 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.703 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:55.637 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.737 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:08:55.670 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.746 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:08:55.647 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.777+0000 7f8143dd7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.777+0000 7f8143dd7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.777+0000 7f8143dd7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:10.777 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.777+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f813c0627b0 con 0x7f813c14be90 2024-03-25T19:50:10.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.842+0000 7f81327fc640 1 -- 172.21.15.130:0/4172250406 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8134052490 con 0x7f813c14be90 2024-03-25T19:50:10.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.842+0000 7f81327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:10.842 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.842+0000 7f81327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:10.842 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:10.843 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 10 librados: watch_flush enter 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 10 librados: watch_flush exit 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 10 client.4594.objecter successfully canceled tick 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 >> v1:172.21.15.163:6800/1217380380 conn(0x7f812006de00 legacy=0x7f81200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.845 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 10 monclient: shutdown 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 >> v1:172.21.15.163:6789/0 conn(0x7f813c14be90 legacy=0x7f813c1e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 shutdown_connections 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 >> 172.21.15.130:0/4172250406 conn(0x7f813c0617a0 msgr2=0x7f813c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 shutdown_connections 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 -- 172.21.15.130:0/4172250406 wait complete. 2024-03-25T19:50:10.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:10.845+0000 7f8143dd7640 1 librados: shutdown 2024-03-25T19:50:10.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:17.663 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:17.778 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.779+0000 7f77be377640 10 monclient: get_monmap_and_config 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.779+0000 7f77be377640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.779+0000 7f77be377640 10 monclient: monmap: 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:17.780086+0000 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:17.780086+0000 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:17.779 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:17.780 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:17.780 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:17.780 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 1 Processor -- start 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 1 -- start start 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient: init 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient(hunting): picked mon.noname-a con 0x7f77b80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient(hunting): picked mon.noname-b con 0x7f77b80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:17.781 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient(hunting): picked mon.noname-c con 0x7f77b814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:17.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80af940 con 0x7f77b80b4d60 2024-03-25T19:50:17.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80b0a70 con 0x7f77b814be90 2024-03-25T19:50:17.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b80b0d50 con 0x7f77b80b1730 2024-03-25T19:50:17.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.782 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.780+0000 7f77be377640 10 monclient(hunting): authenticate will time out at 5493.926550s 2024-03-25T19:50:17.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.791+0000 7f77b77fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f77b80b1730 0x7f77b80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:58482/0 (socket says 172.21.15.130:58482) 2024-03-25T19:50:17.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.791+0000 7f77b77fe640 1 -- 172.21.15.130:0/1957478041 learned_addr learned my addr 172.21.15.130:0/1957478041 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:17.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512755094 0 0) 0x7f77b80b0d50 con 0x7f77b80b1730 2024-03-25T19:50:17.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4598 2024-03-25T19:50:17.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.808+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a0003520 con 0x7f77b80b1730 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662645246 0 0) 0x7f77b80b0a70 con 0x7f77b814be90 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4597 2024-03-25T19:50:17.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.810+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b80b0d50 con 0x7f77b814be90 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2475841286 0 0) 0x7f77b80af940 con 0x7f77b80b4d60 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4602 2024-03-25T19:50:17.821 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.822+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b80b0a70 con 0x7f77b80b4d60 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1879475062 0 0) 0x7f77b80b0d50 con 0x7f77b814be90 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 >> v1:172.21.15.130:6790/0 conn(0x7f77b80b1730 legacy=0x7f77b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 >> v1:172.21.15.130:6789/0 conn(0x7f77b80b4d60 legacy=0x7f77b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 monclient: found mon.noname-c 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b80b1030 con 0x7f77b814be90 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.836 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f77b00019f0 con 0x7f77b814be90 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77b6ffd640 10 monclient: dump: 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.837 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77be377640 5 monclient: authenticate success, global_id 4597 2024-03-25T19:50:17.838 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.836+0000 7f77be377640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:17.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77b0001b50 con 0x7f77b814be90 2024-03-25T19:50:17.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 1 -- 172.21.15.130:0/1957478041 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f77b0001f60 con 0x7f77b814be90 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77be377640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77be377640 4 set_mon_vals no callback set 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.850+0000 7f77b6ffd640 10 monclient: dump: 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:17.850 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:17.851 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:17.851 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:17.851 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 10 monclient: shutdown 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 1 -- 172.21.15.130:0/1957478041 >> v1:172.21.15.163:6789/0 conn(0x7f77b814be90 legacy=0x7f77b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 1 -- 172.21.15.130:0/1957478041 shutdown_connections 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 1 -- 172.21.15.130:0/1957478041 >> 172.21.15.130:0/1957478041 conn(0x7f77b80617a0 msgr2=0x7f77b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 1 -- 172.21.15.130:0/1957478041 shutdown_connections 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 1 -- 172.21.15.130:0/1957478041 wait complete. 2024-03-25T19:50:17.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 10 monclient: build_initial_monmap 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.851+0000 7f77be377640 10 monclient: monmap: 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 librados: starting msgr at 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 librados: starting objecter 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 Processor -- start 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 -- start start 2024-03-25T19:50:17.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 librados: setting wanted keys 2024-03-25T19:50:17.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 1 librados: calling monclient init 2024-03-25T19:50:17.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.852+0000 7f77be377640 10 monclient: init 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient(hunting): picked mon.noname-c con 0x7f77b80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient(hunting): picked mon.noname-a con 0x7f77b80b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient(hunting): picked mon.noname-b con 0x7f77b814be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b81422f0 con 0x7f77b80b4d60 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b813d490 con 0x7f77b814be90 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77b8143290 con 0x7f77b80b1730 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:17.855 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.853+0000 7f77be377640 10 monclient(hunting): authenticate will time out at 5493.999511s 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.863+0000 7f77b77fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f77b80b4d60 0x7f77b8132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:51946/0 (socket says 172.21.15.130:51946) 2024-03-25T19:50:17.863 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.863+0000 7f77b77fe640 1 -- 172.21.15.130:0/699812370 learned_addr learned my addr 172.21.15.130:0/699812370 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:17.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.864+0000 7f77b4ff9640 10 client.?.objecter ms_handle_connect 0x7f77b80b4d60 2024-03-25T19:50:17.864 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.864+0000 7f77b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.866+0000 7f77b4ff9640 10 client.?.objecter ms_handle_connect 0x7f77b814be90 2024-03-25T19:50:17.866 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.866+0000 7f77b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197780922 0 0) 0x7f77b81422f0 con 0x7f77b80b4d60 2024-03-25T19:50:17.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T19:50:17.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.870+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4004eb0 con 0x7f77b80b4d60 2024-03-25T19:50:17.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.875+0000 7f77b4ff9640 10 client.?.objecter ms_handle_connect 0x7f77b80b1730 2024-03-25T19:50:17.874 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.875+0000 7f77b4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545279794 0 0) 0x7f77b813d490 con 0x7f77b814be90 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 10 monclient(hunting): my global_id is 4600 2024-03-25T19:50:17.883 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.883+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b81422f0 con 0x7f77b814be90 2024-03-25T19:50:17.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 169507233 0 0) 0x7f77a4004eb0 con 0x7f77b80b4d60 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b813d490 con 0x7f77b80b4d60 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f77ac001950 con 0x7f77b80b4d60 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.894 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:17.895 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:17.895 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:17.895 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:17.895 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.895 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.894+0000 7f77b4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1893258780 0 0) 0x7f77b8143290 con 0x7f77b80b1730 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 10 monclient(hunting): my global_id is 4601 2024-03-25T19:50:17.896 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.896+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a4004eb0 con 0x7f77b80b1730 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.915+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325037499 0 0) 0x7f77b81422f0 con 0x7f77b814be90 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.915+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77b8143290 con 0x7f77b814be90 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.915+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f77b0001e60 con 0x7f77b814be90 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.916+0000 7f77b4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.916+0000 7f77b4ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:17.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.916+0000 7f77b4ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:17.916 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1348494151 0 0) 0x7f77b813d490 con 0x7f77b80b4d60 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 >> v1:172.21.15.130:6790/0 conn(0x7f77b80b1730 legacy=0x7f77b8141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 >> v1:172.21.15.163:6789/0 conn(0x7f77b814be90 legacy=0x7f77b813dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 1 monclient: found mon.a 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b813d6e0 con 0x7f77b80b4d60 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77be377640 5 monclient: authenticate success, global_id 4605 2024-03-25T19:50:17.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77be377640 10 monclient: _renew_subs 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77be377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.920+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b8141100 con 0x7f77b80b4d60 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.921+0000 7f77be377640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.921+0000 7f77be377640 10 monclient: _renew_subs 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.921+0000 7f77be377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.921+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b8133610 con 0x7f77b80b4d60 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.921+0000 7f77be377640 1 librados: init done 2024-03-25T19:50:17.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.922+0000 7f77be377640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.922+0000 7f77be377640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:17.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.922+0000 7f77be377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:17.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:17.922+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b8141450 con 0x7f77b80b4d60 2024-03-25T19:50:18.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.147+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77ac0047e0 con 0x7f77b80b4d60 2024-03-25T19:50:18.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.147+0000 7f77b4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:18.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.147+0000 7f77b67fc640 4 set_mon_vals no callback set 2024-03-25T19:50:18.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.230+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f77ac004af0 con 0x7f77b80b4d60 2024-03-25T19:50:18.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.230+0000 7f77b4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:18.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.230+0000 7f77b4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.230+0000 7f77b4ff9640 10 monclient: dump: 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:18.230 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:18.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.231+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f77ac01aac0 con 0x7f77b80b4d60 2024-03-25T19:50:18.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.231+0000 7f77b7fff640 10 monclient: get_auth_request con 0x7f77a406df70 auth_method 0 2024-03-25T19:50:18.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.231+0000 7f77b4ff9640 10 client.4605.objecter ms_handle_connect 0x7f77a406df70 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f77ac0548c0 con 0x7f77b80b4d60 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 10 client.4605.objecter ms_dispatch 0x7f77b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77ac050200 con 0x7f77b80b4d60 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:18.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.236+0000 7f77b4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:18.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.470+0000 7f77be377640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.470+0000 7f77be377640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.470 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.470+0000 7f77be377640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:18.471 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.470+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77b813d870 con 0x7f77b80b4d60 2024-03-25T19:50:18.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.499+0000 7f77b4ff9640 1 -- 172.21.15.130:0/699812370 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f77ac052300 con 0x7f77b80b4d60 2024-03-25T19:50:18.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.499+0000 7f77b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:18.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.499+0000 7f77b4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:18.499 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:18.499 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 10 librados: watch_flush enter 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 10 librados: watch_flush exit 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 10 client.4605.objecter successfully canceled tick 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 >> v1:172.21.15.163:6800/1217380380 conn(0x7f77a406df70 legacy=0x7f77a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 10 monclient: shutdown 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 >> v1:172.21.15.130:6789/0 conn(0x7f77b80b4d60 legacy=0x7f77b8132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 shutdown_connections 2024-03-25T19:50:18.502 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 >> 172.21.15.130:0/699812370 conn(0x7f77b80617a0 msgr2=0x7f77b80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:18.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 shutdown_connections 2024-03-25T19:50:18.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 -- 172.21.15.130:0/699812370 wait complete. 2024-03-25T19:50:18.503 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:18.502+0000 7f77be377640 1 librados: shutdown 2024-03-25T19:50:18.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:25.320 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:25.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.434+0000 7f33d2655640 10 monclient: get_monmap_and_config 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.434+0000 7f33d2655640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.434+0000 7f33d2655640 10 monclient: monmap: 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:25.435264+0000 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:25.435264+0000 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:25.434 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.435 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.435+0000 7f33d2655640 1 Processor -- start 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.435+0000 7f33d2655640 1 -- start start 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.435+0000 7f33d2655640 10 monclient: init 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-a con 0x7f33cc0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-c con 0x7f33cc0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-b con 0x7f33cc14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0af940 con 0x7f33cc0b4d60 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b0a70 con 0x7f33cc0b1730 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc0b0d50 con 0x7f33cc14be90 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.436 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.436+0000 7f33d2655640 10 monclient(hunting): authenticate will time out at 5501.581766s 2024-03-25T19:50:25.440 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.441+0000 7f33cbfff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f33cc0b4d60 0x7f33cc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56620/0 (socket says 172.21.15.130:56620) 2024-03-25T19:50:25.441 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.441+0000 7f33cbfff640 1 -- 172.21.15.130:0/4271401331 learned_addr learned my addr 172.21.15.130:0/4271401331 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099824161 0 0) 0x7f33cc0af940 con 0x7f33cc0b4d60 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 10 monclient(hunting): my global_id is 4608 2024-03-25T19:50:25.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.459+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b4003520 con 0x7f33cc0b4d60 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913767579 0 0) 0x7f33cc0b0d50 con 0x7f33cc14be90 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 10 monclient(hunting): my global_id is 4604 2024-03-25T19:50:25.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.466+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc0af940 con 0x7f33cc14be90 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796459137 0 0) 0x7f33cc0b0a70 con 0x7f33cc0b1730 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 10 monclient(hunting): my global_id is 4603 2024-03-25T19:50:25.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.474+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc0b0d50 con 0x7f33cc0b1730 2024-03-25T19:50:25.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143192732 0 0) 0x7f33cc0b0d50 con 0x7f33cc0b1730 2024-03-25T19:50:25.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 >> v1:172.21.15.130:6790/0 conn(0x7f33cc14be90 legacy=0x7f33cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.497 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 >> v1:172.21.15.130:6789/0 conn(0x7f33cc0b4d60 legacy=0x7f33cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 monclient: found mon.noname-c 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc0b1030 con 0x7f33cc0b1730 2024-03-25T19:50:25.498 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f33c00015b0 con 0x7f33cc0b1730 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33caffd640 10 monclient: dump: 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:25.499 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:25.500 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:25.500 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:25.500 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33d2655640 5 monclient: authenticate success, global_id 4603 2024-03-25T19:50:25.500 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.497+0000 7f33d2655640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33c0001710 con 0x7f33cc0b1730 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 1 -- 172.21.15.130:0/4271401331 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f33c0001b60 con 0x7f33cc0b1730 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33caffd640 10 monclient: dump: 2024-03-25T19:50:25.542 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33d2655640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:25.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.542+0000 7f33d2655640 4 set_mon_vals no callback set 2024-03-25T19:50:25.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 10 monclient: shutdown 2024-03-25T19:50:25.544 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 1 -- 172.21.15.130:0/4271401331 >> v1:172.21.15.163:6789/0 conn(0x7f33cc0b1730 legacy=0x7f33cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 1 -- 172.21.15.130:0/4271401331 shutdown_connections 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 1 -- 172.21.15.130:0/4271401331 >> 172.21.15.130:0/4271401331 conn(0x7f33cc0617a0 msgr2=0x7f33cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 1 -- 172.21.15.130:0/4271401331 shutdown_connections 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.543+0000 7f33d2655640 1 -- 172.21.15.130:0/4271401331 wait complete. 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.544+0000 7f33d2655640 10 monclient: build_initial_monmap 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.544+0000 7f33d2655640 10 monclient: monmap: 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:25.545 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.544+0000 7f33d2655640 1 librados: starting msgr at 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.544+0000 7f33d2655640 1 librados: starting objecter 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.544+0000 7f33d2655640 1 Processor -- start 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 -- start start 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 librados: setting wanted keys 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 librados: calling monclient init 2024-03-25T19:50:25.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient: init 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-b con 0x7f33cc0b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-c con 0x7f33cc0b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient(hunting): picked mon.noname-a con 0x7f33cc14be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc137fe0 con 0x7f33cc14be90 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc12f7b0 con 0x7f33cc0b1730 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33cc1374b0 con 0x7f33cc0b4d60 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:25.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.545+0000 7f33d2655640 10 monclient(hunting): authenticate will time out at 5501.691593s 2024-03-25T19:50:25.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.562+0000 7f33d0bf2640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f33cc14be90 0x7f33cc12f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56682/0 (socket says 172.21.15.130:56682) 2024-03-25T19:50:25.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.562+0000 7f33d0bf2640 1 -- 172.21.15.130:0/1433945283 learned_addr learned my addr 172.21.15.130:0/1433945283 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:25.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.563+0000 7f33c8ff9640 10 client.?.objecter ms_handle_connect 0x7f33cc14be90 2024-03-25T19:50:25.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.563+0000 7f33c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.571+0000 7f33c8ff9640 10 client.?.objecter ms_handle_connect 0x7f33cc0b1730 2024-03-25T19:50:25.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.571+0000 7f33c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.575+0000 7f33c8ff9640 10 client.?.objecter ms_handle_connect 0x7f33cc0b4d60 2024-03-25T19:50:25.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.575+0000 7f33c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:25.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115009987 0 0) 0x7f33cc137fe0 con 0x7f33cc14be90 2024-03-25T19:50:25.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): my global_id is 4611 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8004eb0 con 0x7f33cc14be90 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407213232 0 0) 0x7f33cc1374b0 con 0x7f33cc0b4d60 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.582+0000 7f33c8ff9640 10 monclient(hunting): my global_id is 4607 2024-03-25T19:50:25.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.583+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33cc137fe0 con 0x7f33cc0b4d60 2024-03-25T19:50:25.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2419955675 0 0) 0x7f33b8004eb0 con 0x7f33cc14be90 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33cc1374b0 con 0x7f33cc14be90 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f33c4001d30 con 0x7f33cc14be90 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 10 monclient(hunting): dump: 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:25.591 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.591+0000 7f33c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:25.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133666828 0 0) 0x7f33cc12f7b0 con 0x7f33cc0b1730 2024-03-25T19:50:25.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:50:25.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:25.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:25.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 10 monclient(hunting): my global_id is 4606 2024-03-25T19:50:25.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.605+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33b8004eb0 con 0x7f33cc0b1730 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2787402731 0 0) 0x7f33cc1374b0 con 0x7f33cc14be90 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 >> v1:172.21.15.130:6790/0 conn(0x7f33cc0b4d60 legacy=0x7f33cc12e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 >> v1:172.21.15.163:6789/0 conn(0x7f33cc0b1730 legacy=0x7f33cc1378d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 1 monclient: found mon.a 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc145620 con 0x7f33cc14be90 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 5 monclient: authenticate success, global_id 4611 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 10 monclient: _renew_subs 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33cc145830 con 0x7f33cc14be90 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 10 monclient: _renew_subs 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:25.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33cc13e270 con 0x7f33cc14be90 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.610+0000 7f33d2655640 1 librados: init done 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.611+0000 7f33d2655640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.611+0000 7f33d2655640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.611+0000 7f33d2655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:25.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.611+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33cc0b5e10 con 0x7f33cc14be90 2024-03-25T19:50:25.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f33c4003970 con 0x7f33cc14be90 2024-03-25T19:50:25.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:25.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f33c4003d00 con 0x7f33cc14be90 2024-03-25T19:50:25.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:25.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33c8ff9640 10 monclient: dump: 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.921+0000 7f33ca7fc640 4 set_mon_vals no callback set 2024-03-25T19:50:25.922 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.922+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f33c4003f20 con 0x7f33cc14be90 2024-03-25T19:50:25.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.923+0000 7f33cbfff640 10 monclient: get_auth_request con 0x7f33b806de00 auth_method 0 2024-03-25T19:50:25.923 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.923+0000 7f33c8ff9640 10 client.4611.objecter ms_handle_connect 0x7f33b806de00 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f33c4054a10 con 0x7f33cc14be90 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 10 client.4611.objecter ms_dispatch 0x7f33cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33c4050520 con 0x7f33cc14be90 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:25.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:25.930+0000 7f33c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:26.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.158+0000 7f33d2655640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.158+0000 7f33d2655640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.158+0000 7f33d2655640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:26.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.158+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f33cc0627b0 con 0x7f33cc14be90 2024-03-25T19:50:26.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.197+0000 7f33c8ff9640 1 -- 172.21.15.130:0/1433945283 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f33c4052450 con 0x7f33cc14be90 2024-03-25T19:50:26.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.197+0000 7f33c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:26.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.197+0000 7f33c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:26.197 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:26.197 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:26.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 10 librados: watch_flush enter 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 10 librados: watch_flush exit 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 10 client.4611.objecter successfully canceled tick 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 >> v1:172.21.15.163:6800/1217380380 conn(0x7f33b806de00 legacy=0x7f33b80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 10 monclient: shutdown 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.199+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 >> v1:172.21.15.130:6789/0 conn(0x7f33cc14be90 legacy=0x7f33cc12f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.200+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 shutdown_connections 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.200+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 >> 172.21.15.130:0/1433945283 conn(0x7f33cc0617a0 msgr2=0x7f33cc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.200+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 shutdown_connections 2024-03-25T19:50:26.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.200+0000 7f33d2655640 1 -- 172.21.15.130:0/1433945283 wait complete. 2024-03-25T19:50:26.201 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:26.200+0000 7f33d2655640 1 librados: shutdown 2024-03-25T19:50:26.212 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:30.498 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:15.431 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.545 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:15.478 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.567 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:15.500 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.617 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:15.518 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.617 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:15.551 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.624 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:15.525 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.624 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:15.557 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.654 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:15.587 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:30.668 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:15.569 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:33.019 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.135+0000 7f40a4663640 10 monclient: get_monmap_and_config 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.135+0000 7f40a4663640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.135+0000 7f40a4663640 10 monclient: monmap: 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:33.136309+0000 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:33.136309+0000 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.135 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.136+0000 7f40a4663640 1 Processor -- start 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.136+0000 7f40a4663640 1 -- start start 2024-03-25T19:50:33.136 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.136+0000 7f40a4663640 10 monclient: init 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-c con 0x7f409c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-a con 0x7f409c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-b con 0x7f409c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c142520 con 0x7f409c13f710 2024-03-25T19:50:33.137 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0b28a0 con 0x7f409c0a98f0 2024-03-25T19:50:33.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0b2a30 con 0x7f409c1430c0 2024-03-25T19:50:33.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.138 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.137+0000 7f40a4663640 10 monclient(hunting): authenticate will time out at 5509.283465s 2024-03-25T19:50:33.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.145+0000 7f40a23ff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f409c0a98f0 0x7f409c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:41120/0 (socket says 172.21.15.130:41120) 2024-03-25T19:50:33.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.145+0000 7f40a23ff640 1 -- 172.21.15.130:0/625041099 learned_addr learned my addr 172.21.15.130:0/625041099 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:33.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691710258 0 0) 0x7f409c0b28a0 con 0x7f409c0a98f0 2024-03-25T19:50:33.155 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 10 monclient(hunting): my global_id is 4609 2024-03-25T19:50:33.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.156+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4088003520 con 0x7f409c0a98f0 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347910097 0 0) 0x7f409c0b2a30 con 0x7f409c1430c0 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 10 monclient(hunting): my global_id is 4610 2024-03-25T19:50:33.158 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.158+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c0b28a0 con 0x7f409c1430c0 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1259232555 0 0) 0x7f409c0b28a0 con 0x7f409c1430c0 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 >> v1:172.21.15.163:6789/0 conn(0x7f409c0a98f0 legacy=0x7f409c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 >> v1:172.21.15.130:6789/0 conn(0x7f409c13f710 legacy=0x7f409c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 monclient: found mon.noname-b 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.163 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0b44c0 con 0x7f409c1430c0 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f40940019f0 con 0x7f409c1430c0 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a13fd640 10 monclient: dump: 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:33.164 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a4663640 5 monclient: authenticate success, global_id 4610 2024-03-25T19:50:33.165 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.163+0000 7f40a4663640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:33.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4094001b50 con 0x7f409c1430c0 2024-03-25T19:50:33.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 1 -- 172.21.15.130:0/625041099 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f4094001f20 con 0x7f409c1430c0 2024-03-25T19:50:33.172 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:33.173 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a13fd640 10 monclient: dump: 2024-03-25T19:50:33.173 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:33.173 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:33.173 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:33.174 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a4663640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:33.175 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.172+0000 7f40a4663640 4 set_mon_vals no callback set 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 10 monclient: shutdown 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 1 -- 172.21.15.130:0/625041099 >> v1:172.21.15.130:6790/0 conn(0x7f409c1430c0 legacy=0x7f409c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 1 -- 172.21.15.130:0/625041099 shutdown_connections 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 1 -- 172.21.15.130:0/625041099 >> 172.21.15.130:0/625041099 conn(0x7f409c0b7ea0 msgr2=0x7f409c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.173+0000 7f40a4663640 1 -- 172.21.15.130:0/625041099 shutdown_connections 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.174+0000 7f40a4663640 1 -- 172.21.15.130:0/625041099 wait complete. 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.174+0000 7f40a4663640 10 monclient: build_initial_monmap 2024-03-25T19:50:33.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.174+0000 7f40a4663640 10 monclient: monmap: 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 librados: starting msgr at 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 librados: starting objecter 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 Processor -- start 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 -- start start 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 librados: setting wanted keys 2024-03-25T19:50:33.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 1 librados: calling monclient init 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.175+0000 7f40a4663640 10 monclient: init 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-a con 0x7f409c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-c con 0x7f409c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient(hunting): picked mon.noname-b con 0x7f409c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0b02b0 con 0x7f409c0a98f0 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0b0500 con 0x7f409c1430c0 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f409c0aea50 con 0x7f409c13f710 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:33.180 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.176+0000 7f40a4663640 10 monclient(hunting): authenticate will time out at 5509.321770s 2024-03-25T19:50:33.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.186+0000 7f40a23ff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f409c0a98f0 0x7f409c0ae180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:46094/0 (socket says 172.21.15.130:46094) 2024-03-25T19:50:33.185 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.186+0000 7f40a23ff640 1 -- 172.21.15.130:0/369718251 learned_addr learned my addr 172.21.15.130:0/369718251 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:33.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.187+0000 7f4092ffd640 10 client.?.objecter ms_handle_connect 0x7f409c0a98f0 2024-03-25T19:50:33.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.187+0000 7f4092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.187+0000 7f4092ffd640 10 client.?.objecter ms_handle_connect 0x7f409c13f710 2024-03-25T19:50:33.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.187+0000 7f4092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.190 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.191+0000 7f4092ffd640 10 client.?.objecter ms_handle_connect 0x7f409c1430c0 2024-03-25T19:50:33.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.191+0000 7f4092ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:33.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2917892353 0 0) 0x7f409c0aea50 con 0x7f409c13f710 2024-03-25T19:50:33.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 10 monclient(hunting): my global_id is 4613 2024-03-25T19:50:33.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.194+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080004eb0 con 0x7f409c13f710 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2394424090 0 0) 0x7f409c0b02b0 con 0x7f409c0a98f0 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 10 monclient(hunting): my global_id is 4617 2024-03-25T19:50:33.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.213+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f409c0aea50 con 0x7f409c0a98f0 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650631735 0 0) 0x7f4080004eb0 con 0x7f409c13f710 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f409c0b02b0 con 0x7f409c13f710 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f408c0018c0 con 0x7f409c13f710 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 10 monclient(hunting): dump: 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.216 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.216+0000 7f4092ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839619100 0 0) 0x7f409c0b0500 con 0x7f409c1430c0 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 10 monclient(hunting): my global_id is 4612 2024-03-25T19:50:33.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.219+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4080004eb0 con 0x7f409c1430c0 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2006617239 0 0) 0x7f409c0b02b0 con 0x7f409c13f710 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 >> v1:172.21.15.163:6789/0 conn(0x7f409c1430c0 legacy=0x7f409c133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 >> v1:172.21.15.130:6789/0 conn(0x7f409c0a98f0 legacy=0x7f409c0ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 1 monclient: found mon.c 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:33.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0af680 con 0x7f409c13f710 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 10 monclient: _finish_auth 0 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f4092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 5 monclient: authenticate success, global_id 4613 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 10 monclient: _renew_subs 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f409c133780 con 0x7f409c13f710 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 10 monclient: _renew_subs 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f409c0aba90 con 0x7f409c13f710 2024-03-25T19:50:33.230 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.229+0000 7f40a4663640 1 librados: init done 2024-03-25T19:50:33.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.230+0000 7f40a4663640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.230+0000 7f40a4663640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.230+0000 7f40a4663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.230+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f409c133b20 con 0x7f409c13f710 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f408c0035f0 con 0x7f409c13f710 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f408c003860 con 0x7f409c13f710 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f4092ffd640 10 monclient: dump: 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:33.472 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:33.473 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.472+0000 7f40a0bfc640 4 set_mon_vals no callback set 2024-03-25T19:50:33.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.473+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f408c003a80 con 0x7f409c13f710 2024-03-25T19:50:33.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.474+0000 7f40a23ff640 10 monclient: get_auth_request con 0x7f408006de00 auth_method 0 2024-03-25T19:50:33.474 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.474+0000 7f4092ffd640 10 client.4613.objecter ms_handle_connect 0x7f408006de00 2024-03-25T19:50:33.477 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.477+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f408c0533f0 con 0x7f409c13f710 2024-03-25T19:50:33.477 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.477+0000 7f4092ffd640 10 client.4613.objecter ms_dispatch 0x7f409c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.477+0000 7f4092ffd640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.477+0000 7f4092ffd640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.478+0000 7f4092ffd640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.478+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f408c04ef00 con 0x7f409c13f710 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.478+0000 7f4092ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:33.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.478+0000 7f4092ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:33.712 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.712+0000 7f40a4663640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.712 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.712+0000 7f40a4663640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.712 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.712+0000 7f40a4663640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:50:33.712 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.712+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f409c0aebe0 con 0x7f409c13f710 2024-03-25T19:50:33.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.732+0000 7f4092ffd640 1 -- 172.21.15.130:0/369718251 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f408c050e30 con 0x7f409c13f710 2024-03-25T19:50:33.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.732+0000 7f4092ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:33.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.732+0000 7f4092ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:33.732 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:33.733 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.734+0000 7f40a4663640 10 librados: watch_flush enter 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 10 librados: watch_flush exit 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 10 client.4613.objecter successfully canceled tick 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 >> v1:172.21.15.163:6800/1217380380 conn(0x7f408006de00 legacy=0x7f40800702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 10 monclient: shutdown 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 >> v1:172.21.15.130:6790/0 conn(0x7f409c13f710 legacy=0x7f409c0aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:33.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 shutdown_connections 2024-03-25T19:50:33.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 >> 172.21.15.130:0/369718251 conn(0x7f409c0b7ea0 msgr2=0x7f409c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:33.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 shutdown_connections 2024-03-25T19:50:33.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 -- 172.21.15.130:0/369718251 wait complete. 2024-03-25T19:50:33.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:33.735+0000 7f40a4663640 1 librados: shutdown 2024-03-25T19:50:33.746 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:40.553 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.673+0000 7f8d6c086640 10 monclient: get_monmap_and_config 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.673+0000 7f8d6c086640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.673+0000 7f8d6c086640 10 monclient: monmap: 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:40.674749+0000 2024-03-25T19:50:40.673 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:40.674749+0000 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:40.674 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.674+0000 7f8d6c086640 1 Processor -- start 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.674+0000 7f8d6c086640 1 -- start start 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.674+0000 7f8d6c086640 10 monclient: init 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-b con 0x7f8d640b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-c con 0x7f8d640b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:40.675 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-a con 0x7f8d6414be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:40.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640af940 con 0x7f8d6414be90 2024-03-25T19:50:40.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640b0a70 con 0x7f8d640b1730 2024-03-25T19:50:40.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d640b0d50 con 0x7f8d640b4d60 2024-03-25T19:50:40.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.676 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.675+0000 7f8d6c086640 10 monclient(hunting): authenticate will time out at 5516.821279s 2024-03-25T19:50:40.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.685+0000 7f8d6a623640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f8d6414be90 0x7f8d6414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43140/0 (socket says 172.21.15.130:43140) 2024-03-25T19:50:40.685 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.685+0000 7f8d6a623640 1 -- 172.21.15.130:0/3362754753 learned_addr learned my addr 172.21.15.130:0/3362754753 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146722115 0 0) 0x7f8d640af940 con 0x7f8d6414be90 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 10 monclient(hunting): my global_id is 4620 2024-03-25T19:50:40.693 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.693+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d50003520 con 0x7f8d6414be90 2024-03-25T19:50:40.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3145638978 0 0) 0x7f8d50003520 con 0x7f8d6414be90 2024-03-25T19:50:40.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 >> v1:172.21.15.130:6790/0 conn(0x7f8d640b4d60 legacy=0x7f8d6413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:50:40.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 >> v1:172.21.15.163:6789/0 conn(0x7f8d640b1730 legacy=0x7f8d640b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 monclient: found mon.noname-a 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d640b1030 con 0x7f8d6414be90 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8d5c001a10 con 0x7f8d6414be90 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.703+0000 7f8d68e20640 10 monclient: dump: 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.704 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.704+0000 7f8d6c086640 5 monclient: authenticate success, global_id 4620 2024-03-25T19:50:40.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.704+0000 7f8d6c086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d5c001be0 con 0x7f8d6414be90 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 1 -- 172.21.15.130:0/3362754753 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8d5c002030 con 0x7f8d6414be90 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d68e20640 10 monclient: dump: 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.716 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d6c086640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:40.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.716+0000 7f8d6c086640 4 set_mon_vals no callback set 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 10 monclient: shutdown 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 1 -- 172.21.15.130:0/3362754753 >> v1:172.21.15.130:6789/0 conn(0x7f8d6414be90 legacy=0x7f8d6414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 1 -- 172.21.15.130:0/3362754753 shutdown_connections 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 1 -- 172.21.15.130:0/3362754753 >> 172.21.15.130:0/3362754753 conn(0x7f8d640617a0 msgr2=0x7f8d64061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 1 -- 172.21.15.130:0/3362754753 shutdown_connections 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.717+0000 7f8d6c086640 1 -- 172.21.15.130:0/3362754753 wait complete. 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 10 monclient: build_initial_monmap 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 10 monclient: monmap: 2024-03-25T19:50:40.719 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 librados: starting msgr at 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 librados: starting objecter 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 Processor -- start 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 -- start start 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 librados: setting wanted keys 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 1 librados: calling monclient init 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.718+0000 7f8d6c086640 10 monclient: init 2024-03-25T19:50:40.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-a con 0x7f8d640b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-c con 0x7f8d640b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient(hunting): picked mon.noname-b con 0x7f8d6414be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64139020 con 0x7f8d640b1730 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64145590 con 0x7f8d6414be90 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d64145720 con 0x7f8d640b4d60 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:40.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.719+0000 7f8d6c086640 10 monclient(hunting): authenticate will time out at 5516.865018s 2024-03-25T19:50:40.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.733+0000 7f8d69e22640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f8d640b1730 0x7f8d641d7a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43180/0 (socket says 172.21.15.130:43180) 2024-03-25T19:50:40.732 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.733+0000 7f8d69e22640 1 -- 172.21.15.130:0/674754671 learned_addr learned my addr 172.21.15.130:0/674754671 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:40.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.736+0000 7f8d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d640b1730 2024-03-25T19:50:40.735 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.736+0000 7f8d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.740+0000 7f8d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d6414be90 2024-03-25T19:50:40.739 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.740+0000 7f8d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.749+0000 7f8d5a7fc640 10 client.?.objecter ms_handle_connect 0x7f8d640b4d60 2024-03-25T19:50:40.749 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.749+0000 7f8d5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:40.759 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 491415289 0 0) 0x7f8d64145590 con 0x7f8d6414be90 2024-03-25T19:50:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 10 monclient(hunting): my global_id is 4618 2024-03-25T19:50:40.760 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.760+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d48004eb0 con 0x7f8d6414be90 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426235481 0 0) 0x7f8d64139020 con 0x7f8d640b1730 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 10 monclient(hunting): my global_id is 4623 2024-03-25T19:50:40.765 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.765+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64145590 con 0x7f8d640b1730 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1630864745 0 0) 0x7f8d64145720 con 0x7f8d640b4d60 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 10 monclient(hunting): my global_id is 4616 2024-03-25T19:50:40.768 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.768+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d64139020 con 0x7f8d640b4d60 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 313545730 0 0) 0x7f8d48004eb0 con 0x7f8d6414be90 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d64145720 con 0x7f8d6414be90 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8d5c001ef0 con 0x7f8d6414be90 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 10 monclient(hunting): dump: 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.771 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.771+0000 7f8d5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163025315 0 0) 0x7f8d64145590 con 0x7f8d640b1730 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d48004eb0 con 0x7f8d640b1730 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8d60004000 con 0x7f8d640b1730 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.791+0000 7f8d5a7fc640 10 monclient(hunting): dump: 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:40.791 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:40.792 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:40.792 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:40.792 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 889929832 0 0) 0x7f8d64145720 con 0x7f8d6414be90 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 >> v1:172.21.15.130:6790/0 conn(0x7f8d640b4d60 legacy=0x7f8d641d8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 >> v1:172.21.15.130:6789/0 conn(0x7f8d640b1730 legacy=0x7f8d641d7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 1 monclient: found mon.b 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:40.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d641458b0 con 0x7f8d6414be90 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 5 monclient: authenticate success, global_id 4618 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 10 monclient: _renew_subs 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d64139440 con 0x7f8d6414be90 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 10 monclient: _renew_subs 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6413e0e0 con 0x7f8d6414be90 2024-03-25T19:50:40.806 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.805+0000 7f8d6c086640 1 librados: init done 2024-03-25T19:50:40.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.806+0000 7f8d6c086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:40.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.806+0000 7f8d6c086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:40.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.806+0000 7f8d6c086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:40.807 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:40.806+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d64138e60 con 0x7f8d6414be90 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d5c002090 con 0x7f8d6414be90 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8d5c004fb0 con 0x7f8d6414be90 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5a7fc640 10 monclient: dump: 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:41.098 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.098+0000 7f8d5bfff640 4 set_mon_vals no callback set 2024-03-25T19:50:41.099 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.099+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8d5c01af80 con 0x7f8d6414be90 2024-03-25T19:50:41.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.099+0000 7f8d69e22640 10 monclient: get_auth_request con 0x7f8d4806dec0 auth_method 0 2024-03-25T19:50:41.100 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.100+0000 7f8d5a7fc640 10 client.4618.objecter ms_handle_connect 0x7f8d4806dec0 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f8d5c054aa0 con 0x7f8d6414be90 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 10 client.4618.objecter ms_dispatch 0x7f8d64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d5c0505b0 con 0x7f8d6414be90 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:41.106 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.106+0000 7f8d5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:41.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.338+0000 7f8d6c086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.338+0000 7f8d6c086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.338+0000 7f8d6c086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:41.338 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.338+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d6413e430 con 0x7f8d6414be90 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.380+0000 7f8d5a7fc640 1 -- 172.21.15.130:0/674754671 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f8d5c0524e0 con 0x7f8d6414be90 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.380+0000 7f8d5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.380+0000 7f8d5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:41.380 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 10 librados: watch_flush enter 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 10 librados: watch_flush exit 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 10 client.4618.objecter successfully canceled tick 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 >> v1:172.21.15.163:6800/1217380380 conn(0x7f8d4806dec0 legacy=0x7f8d48070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 10 monclient: shutdown 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 >> v1:172.21.15.163:6789/0 conn(0x7f8d6414be90 legacy=0x7f8d641e24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 shutdown_connections 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 >> 172.21.15.130:0/674754671 conn(0x7f8d640617a0 msgr2=0x7f8d64138440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 shutdown_connections 2024-03-25T19:50:41.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 -- 172.21.15.130:0/674754671 wait complete. 2024-03-25T19:50:41.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:41.383+0000 7f8d6c086640 1 librados: shutdown 2024-03-25T19:50:41.395 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:48.201 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.315+0000 7f589df72640 10 monclient: get_monmap_and_config 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.315+0000 7f589df72640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.315+0000 7f589df72640 10 monclient: monmap: 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:48.316381+0000 2024-03-25T19:50:48.315 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:48.316381+0000 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:48.316 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.316+0000 7f589df72640 1 Processor -- start 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.316+0000 7f589df72640 1 -- start start 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.316+0000 7f589df72640 10 monclient: init 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient(hunting): picked mon.noname-a con 0x7f58980a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient(hunting): picked mon.noname-c con 0x7f589813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient(hunting): picked mon.noname-b con 0x7f58981430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5898142520 con 0x7f58980a98f0 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58980b28a0 con 0x7f589813f710 2024-03-25T19:50:48.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58980b2a30 con 0x7f58981430c0 2024-03-25T19:50:48.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.317+0000 7f589df72640 10 monclient(hunting): authenticate will time out at 5524.462985s 2024-03-25T19:50:48.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.328+0000 7f58977fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f58980a98f0 0x7f58980a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43588/0 (socket says 172.21.15.130:43588) 2024-03-25T19:50:48.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.328+0000 7f58977fe640 1 -- 172.21.15.130:0/3251000591 learned_addr learned my addr 172.21.15.130:0/3251000591 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:48.334 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4185071869 0 0) 0x7f5898142520 con 0x7f58980a98f0 2024-03-25T19:50:48.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 10 monclient(hunting): my global_id is 4626 2024-03-25T19:50:48.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.335+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003520 con 0x7f58980a98f0 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515383414 0 0) 0x7f58980b28a0 con 0x7f589813f710 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 10 monclient(hunting): my global_id is 4621 2024-03-25T19:50:48.339 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.339+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5898142520 con 0x7f589813f710 2024-03-25T19:50:48.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058929004 0 0) 0x7f58980b2a30 con 0x7f58981430c0 2024-03-25T19:50:48.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.350 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 10 monclient(hunting): my global_id is 4619 2024-03-25T19:50:48.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.351+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58980b28a0 con 0x7f58981430c0 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1948514172 0 0) 0x7f5880003520 con 0x7f58980a98f0 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 >> v1:172.21.15.130:6790/0 conn(0x7f58981430c0 legacy=0x7f589814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 >> v1:172.21.15.163:6789/0 conn(0x7f589813f710 legacy=0x7f58981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 monclient: found mon.noname-a 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.355 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58980b44c0 con 0x7f58980a98f0 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5888001690 con 0x7f58980a98f0 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f58967fc640 10 monclient: dump: 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:48.356 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f589df72640 5 monclient: authenticate success, global_id 4626 2024-03-25T19:50:48.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.355+0000 7f589df72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:48.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58880017f0 con 0x7f58980a98f0 2024-03-25T19:50:48.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 1 -- 172.21.15.130:0/3251000591 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5888001c00 con 0x7f58980a98f0 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f589df72640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f589df72640 4 set_mon_vals no callback set 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.384+0000 7f58967fc640 10 monclient: dump: 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:48.384 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 10 monclient: shutdown 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 1 -- 172.21.15.130:0/3251000591 >> v1:172.21.15.130:6789/0 conn(0x7f58980a98f0 legacy=0x7f58980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 1 -- 172.21.15.130:0/3251000591 shutdown_connections 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 1 -- 172.21.15.130:0/3251000591 >> 172.21.15.130:0/3251000591 conn(0x7f58980b7ea0 msgr2=0x7f58980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 1 -- 172.21.15.130:0/3251000591 shutdown_connections 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.385+0000 7f589df72640 1 -- 172.21.15.130:0/3251000591 wait complete. 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 10 monclient: build_initial_monmap 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 10 monclient: monmap: 2024-03-25T19:50:48.386 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 librados: starting msgr at 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 librados: starting objecter 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 Processor -- start 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 -- start start 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 librados: setting wanted keys 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 1 librados: calling monclient init 2024-03-25T19:50:48.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.386+0000 7f589df72640 10 monclient: init 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient(hunting): picked mon.noname-a con 0x7f58980a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient(hunting): picked mon.noname-b con 0x7f589813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient(hunting): picked mon.noname-c con 0x7f58981430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58981db050 con 0x7f58980a98f0 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58981dab70 con 0x7f589813f710 2024-03-25T19:50:48.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58981dad00 con 0x7f58981430c0 2024-03-25T19:50:48.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:48.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.387+0000 7f589df72640 10 monclient(hunting): authenticate will time out at 5524.533077s 2024-03-25T19:50:48.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.391+0000 7f58977fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f58980a98f0 0x7f58981db4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43648/0 (socket says 172.21.15.130:43648) 2024-03-25T19:50:48.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.391+0000 7f58977fe640 1 -- 172.21.15.130:0/1518571370 learned_addr learned my addr 172.21.15.130:0/1518571370 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:48.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.392+0000 7f587ffff640 10 client.?.objecter ms_handle_connect 0x7f58980a98f0 2024-03-25T19:50:48.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.392+0000 7f587ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.399+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4036995932 0 0) 0x7f58981db050 con 0x7f58980a98f0 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.399+0000 7f587ffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.399+0000 7f587ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.399+0000 7f587ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.399+0000 7f587ffff640 10 monclient(hunting): my global_id is 4629 2024-03-25T19:50:48.399 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.400+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5884004eb0 con 0x7f58980a98f0 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.400+0000 7f587ffff640 10 client.?.objecter ms_handle_connect 0x7f589813f710 2024-03-25T19:50:48.400 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.400+0000 7f587ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.403+0000 7f587ffff640 10 client.?.objecter ms_handle_connect 0x7f58981430c0 2024-03-25T19:50:48.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.403+0000 7f587ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562154225 0 0) 0x7f5884004eb0 con 0x7f58980a98f0 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58981db050 con 0x7f58980a98f0 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5888001bc0 con 0x7f58980a98f0 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 10 monclient(hunting): dump: 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.408 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:48.409 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:48.409 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:48.409 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:48.409 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.408+0000 7f587ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2837115840 0 0) 0x7f58981db050 con 0x7f58980a98f0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 >> v1:172.21.15.130:6790/0 conn(0x7f58981430c0 legacy=0x7f58981393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 >> v1:172.21.15.163:6789/0 conn(0x7f589813f710 legacy=0x7f58981d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 1 monclient: found mon.a 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58980b70a0 con 0x7f58980a98f0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:48.416 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f587ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f589df72640 5 monclient: authenticate success, global_id 4629 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f589df72640 10 monclient: _renew_subs 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f589df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.414+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58980b72f0 con 0x7f58980a98f0 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 monclient: _renew_subs 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58981e32a0 con 0x7f58980a98f0 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 1 librados: init done 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.417 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.415+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58980627b0 con 0x7f58980a98f0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58880011f0 con 0x7f58980a98f0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5888004c30 con 0x7f58980a98f0 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:48.720 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f587ffff640 10 monclient: dump: 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.720+0000 7f5895ffb640 4 set_mon_vals no callback set 2024-03-25T19:50:48.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.721+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f5888004e50 con 0x7f58980a98f0 2024-03-25T19:50:48.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.721+0000 7f5896ffd640 10 monclient: get_auth_request con 0x7f588406dc40 auth_method 0 2024-03-25T19:50:48.722 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.722+0000 7f587ffff640 10 client.4629.objecter ms_handle_connect 0x7f588406dc40 2024-03-25T19:50:48.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f58880549c0 con 0x7f58980a98f0 2024-03-25T19:50:48.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 10 client.4629.objecter ms_dispatch 0x7f5898000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:48.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:48.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:48.729 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T19:50:48.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5888050300 con 0x7f58980a98f0 2024-03-25T19:50:48.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:48.730 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.729+0000 7f587ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:48.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.980+0000 7f589df72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:48.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.980+0000 7f589df72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:48.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.980+0000 7f589df72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:50:48.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.980+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5898069f40 con 0x7f58980a98f0 2024-03-25T19:50:48.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.999+0000 7f587ffff640 1 -- 172.21.15.130:0/1518571370 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f5888052400 con 0x7f58980a98f0 2024-03-25T19:50:48.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.999+0000 7f587ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:48.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:48.999+0000 7f587ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:49.000 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:49.000 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:49.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 10 librados: watch_flush enter 2024-03-25T19:50:49.002 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 10 librados: watch_flush exit 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 10 client.4629.objecter successfully canceled tick 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 >> v1:172.21.15.163:6800/1217380380 conn(0x7f588406dc40 legacy=0x7f5884070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 10 monclient: shutdown 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 >> v1:172.21.15.130:6789/0 conn(0x7f58980a98f0 legacy=0x7f58981db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 shutdown_connections 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 >> 172.21.15.130:0/1518571370 conn(0x7f58980b7ea0 msgr2=0x7f5898074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:49.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 shutdown_connections 2024-03-25T19:50:49.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 -- 172.21.15.130:0/1518571370 wait complete. 2024-03-25T19:50:49.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:49.002+0000 7f589df72640 1 librados: shutdown 2024-03-25T19:50:49.015 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:50:50.555 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:35.488 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.611 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:35.544 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.636 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:35.570 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.660 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:35.593 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.664 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:35.597 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.681 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:35.582 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.686 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:35.587 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.696 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:35.629 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:50.714 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:35.615 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:50:55.822 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:50:55.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 10 monclient: get_monmap_and_config 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 10 monclient: build_initial_monmap 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 10 monclient: monmap: 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:50:55.943953+0000 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:50:55.943953+0000 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:50:55.943 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:50:55.944 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 1 Processor -- start 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 1 -- start start 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.943+0000 7f3bef8c8640 10 monclient: init 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f3be80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f3be813f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f3be81430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:55.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be8142520 con 0x7f3be813f710 2024-03-25T19:50:55.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b28a0 con 0x7f3be81430c0 2024-03-25T19:50:55.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80b2a30 con 0x7f3be80a98f0 2024-03-25T19:50:55.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:55.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.944+0000 7f3bef8c8640 10 monclient(hunting): authenticate will time out at 5532.090427s 2024-03-25T19:50:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.949+0000 7f3bece63640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f3be813f710 0x7f3be81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:48480/0 (socket says 172.21.15.130:48480) 2024-03-25T19:50:55.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.949+0000 7f3bece63640 1 -- 172.21.15.130:0/3494452681 learned_addr learned my addr 172.21.15.130:0/3494452681 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2725273391 0 0) 0x7f3be8142520 con 0x7f3be813f710 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 10 monclient(hunting): my global_id is 4632 2024-03-25T19:50:55.961 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.961+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bd4003520 con 0x7f3be813f710 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4142169772 0 0) 0x7f3be80b2a30 con 0x7f3be80a98f0 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 10 monclient(hunting): my global_id is 4625 2024-03-25T19:50:55.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.967+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be8142520 con 0x7f3be80a98f0 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82547755 0 0) 0x7f3be80b28a0 con 0x7f3be81430c0 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 10 monclient(hunting): _negotiate 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 10 monclient(hunting): my global_id is 4627 2024-03-25T19:50:55.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.969+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3be80b2a30 con 0x7f3be81430c0 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1319358056 0 0) 0x7f3bd4003520 con 0x7f3be813f710 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 >> v1:172.21.15.130:6790/0 conn(0x7f3be80a98f0 legacy=0x7f3be80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 >> v1:172.21.15.163:6789/0 conn(0x7f3be81430c0 legacy=0x7f3be814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 10 monclient: _finish_hunting 0 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 monclient: found mon.noname-a 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be80b44c0 con 0x7f3be813f710 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 10 monclient: _finish_auth 0 2024-03-25T19:50:55.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 10 monclient: _check_auth_tickets 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bef8c8640 5 monclient: authenticate success, global_id 4632 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bef8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.979+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bd8001690 con 0x7f3be813f710 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.980+0000 7f3bdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.980+0000 7f3bdffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.980+0000 7f3bdffff640 10 monclient: dump: 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:55.981 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:55.982 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:55.982 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:55.982 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:55.982 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.980+0000 7f3bef8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bd80017f0 con 0x7f3be813f710 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 1 -- 172.21.15.130:0/3494452681 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bd8001c00 con 0x7f3be813f710 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bdffff640 10 monclient: dump: 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:55.995 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bef8c8640 10 monclient: get_monmap_and_config success 2024-03-25T19:50:55.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.995+0000 7f3bef8c8640 4 set_mon_vals no callback set 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 10 monclient: shutdown 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 1 -- 172.21.15.130:0/3494452681 >> v1:172.21.15.130:6789/0 conn(0x7f3be813f710 legacy=0x7f3be81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 1 -- 172.21.15.130:0/3494452681 shutdown_connections 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 1 -- 172.21.15.130:0/3494452681 >> 172.21.15.130:0/3494452681 conn(0x7f3be80b7ea0 msgr2=0x7f3be80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 1 -- 172.21.15.130:0/3494452681 shutdown_connections 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.996+0000 7f3bef8c8640 1 -- 172.21.15.130:0/3494452681 wait complete. 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 10 monclient: build_initial_monmap 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 10 monclient: monmap: 2024-03-25T19:50:55.999 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 librados: starting msgr at 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 librados: starting objecter 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 Processor -- start 2024-03-25T19:50:56.000 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 -- start start 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 librados: setting wanted keys 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 1 librados: calling monclient init 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.997+0000 7f3bef8c8640 10 monclient: init 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient: _reopen_session rank -1 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f3be80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f3be813f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f3be81430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be80abd70 con 0x7f3be813f710 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be81d7710 con 0x7f3be81430c0 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3be81d8340 con 0x7f3be80a98f0 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient(hunting): _renew_subs 2024-03-25T19:50:56.001 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:55.998+0000 7f3bef8c8640 10 monclient(hunting): authenticate will time out at 5532.144218s 2024-03-25T19:50:56.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.004+0000 7f3bede65640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3be81430c0 0x7f3be81d4860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:44494/0 (socket says 172.21.15.130:44494) 2024-03-25T19:50:56.003 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.004+0000 7f3bede65640 1 -- 172.21.15.130:0/1311348279 learned_addr learned my addr 172.21.15.130:0/1311348279 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:50:56.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.005+0000 7f3bddffb640 10 client.?.objecter ms_handle_connect 0x7f3be81430c0 2024-03-25T19:50:56.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.005+0000 7f3bddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709903997 0 0) 0x7f3be81d7710 con 0x7f3be81430c0 2024-03-25T19:50:56.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 10 monclient(hunting): _negotiate 2024-03-25T19:50:56.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:50:56.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:50:56.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 10 monclient(hunting): my global_id is 4630 2024-03-25T19:50:56.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.011+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bcc004eb0 con 0x7f3be81430c0 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.014+0000 7f3bddffb640 10 client.?.objecter ms_handle_connect 0x7f3be80a98f0 2024-03-25T19:50:56.014 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.014+0000 7f3bddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157904995 0 0) 0x7f3bcc004eb0 con 0x7f3be81430c0 2024-03-25T19:50:56.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3be81d7710 con 0x7f3be81430c0 2024-03-25T19:50:56.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3be0001c70 con 0x7f3be81430c0 2024-03-25T19:50:56.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 10 monclient(hunting): dump: 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:56.018 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.017+0000 7f3bddffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:50:56.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.019+0000 7f3bddffb640 10 client.?.objecter ms_handle_connect 0x7f3be813f710 2024-03-25T19:50:56.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.019+0000 7f3bddffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 339188716 0 0) 0x7f3be81d7710 con 0x7f3be81430c0 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 >> v1:172.21.15.130:6790/0 conn(0x7f3be80a98f0 legacy=0x7f3be8137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 >> v1:172.21.15.130:6789/0 conn(0x7f3be813f710 legacy=0x7f3be81d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 10 monclient: _finish_hunting 0 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 1 monclient: found mon.b 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3be81d84d0 con 0x7f3be81430c0 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 10 monclient: _finish_auth 0 2024-03-25T19:50:56.025 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 10 monclient: _check_auth_tickets 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 5 monclient: authenticate success, global_id 4630 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 10 monclient: _renew_subs 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3be81d41e0 con 0x7f3be81430c0 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 10 client.4630.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 10 monclient: _renew_subs 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3be81d5150 con 0x7f3be81430c0 2024-03-25T19:50:56.026 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.025+0000 7f3bef8c8640 1 librados: init done 2024-03-25T19:50:56.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.026+0000 7f3bef8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.026+0000 7f3bef8c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.026+0000 7f3bef8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:56.027 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.026+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3be80627b0 con 0x7f3be81430c0 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3be0003990 con 0x7f3be81430c0 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3be0003da0 con 0x7f3be81430c0 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bddffb640 10 monclient: dump: 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:50:56.330 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:50:56.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.330+0000 7f3bdf7fe640 4 set_mon_vals no callback set 2024-03-25T19:50:56.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.331+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f3be0003fc0 con 0x7f3be81430c0 2024-03-25T19:50:56.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.332+0000 7f3bed664640 10 monclient: get_auth_request con 0x7f3bcc06dbf0 auth_method 0 2024-03-25T19:50:56.332 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.332+0000 7f3bddffb640 10 client.4630.objecter ms_handle_connect 0x7f3bcc06dbf0 2024-03-25T19:50:56.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.335+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f3be0054b50 con 0x7f3be81430c0 2024-03-25T19:50:56.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 10 client.4630.objecter ms_dispatch 0x7f3be8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:50:56.335 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 3 client.4630.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:50:56.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 3 client.4630.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:50:56.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 20 client.4630.objecter dump_active .. 0 homeless 2024-03-25T19:50:56.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3be0050660 con 0x7f3be81430c0 2024-03-25T19:50:56.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:50:56.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.336+0000 7f3bddffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:50:56.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.561+0000 7f3bef8c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.561+0000 7f3bef8c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.561+0000 7f3bef8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:50:56.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.561+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3be81d78a0 con 0x7f3be81430c0 2024-03-25T19:50:56.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.589+0000 7f3bddffb640 1 -- 172.21.15.130:0/1311348279 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f3be0052590 con 0x7f3be81430c0 2024-03-25T19:50:56.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.589+0000 7f3bddffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:50:56.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.589+0000 7f3bddffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:50:56.590 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:50:56.590 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 10 librados: watch_flush enter 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 10 librados: watch_flush exit 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 20 client.4630.objecter shutdown clearing up homeless session... 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 10 client.4630.objecter successfully canceled tick 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 >> v1:172.21.15.163:6800/1217380380 conn(0x7f3bcc06dbf0 legacy=0x7f3bcc0700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 10 monclient: shutdown 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 >> v1:172.21.15.163:6789/0 conn(0x7f3be81430c0 legacy=0x7f3be81d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 shutdown_connections 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 >> 172.21.15.130:0/1311348279 conn(0x7f3be80b7ea0 msgr2=0x7f3be8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:50:56.593 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 shutdown_connections 2024-03-25T19:50:56.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 -- 172.21.15.130:0/1311348279 wait complete. 2024-03-25T19:50:56.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:50:56.592+0000 7f3bef8c8640 1 librados: shutdown 2024-03-25T19:50:56.603 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:03.410 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.525+0000 7fad7a590640 10 monclient: get_monmap_and_config 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.525+0000 7fad7a590640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.525+0000 7fad7a590640 10 monclient: monmap: 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:03.527043+0000 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:03.527043+0000 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.526 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.527 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:03.527 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:03.527 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:03.527 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.526+0000 7fad7a590640 1 Processor -- start 2024-03-25T19:51:03.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.526+0000 7fad7a590640 1 -- start start 2024-03-25T19:51:03.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.526+0000 7fad7a590640 10 monclient: init 2024-03-25T19:51:03.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.528 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-a con 0x7fad740a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-b con 0x7fad7413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-c con 0x7fad741430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad74142520 con 0x7fad740a98f0 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740b28a0 con 0x7fad741430c0 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740b2a30 con 0x7fad7413f710 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.527+0000 7fad7a590640 10 monclient(hunting): authenticate will time out at 5539.673782s 2024-03-25T19:51:03.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.532+0000 7fad73fff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fad740a98f0 0x7fad740a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:42492/0 (socket says 172.21.15.130:42492) 2024-03-25T19:51:03.533 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.532+0000 7fad73fff640 1 -- 172.21.15.130:0/1824519520 learned_addr learned my addr 172.21.15.130:0/1824519520 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:03.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63206729 0 0) 0x7fad74142520 con 0x7fad740a98f0 2024-03-25T19:51:03.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 10 monclient(hunting): my global_id is 4638 2024-03-25T19:51:03.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.550+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad60003520 con 0x7fad740a98f0 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40159399 0 0) 0x7fad740b2a30 con 0x7fad7413f710 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 10 monclient(hunting): my global_id is 4631 2024-03-25T19:51:03.555 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.554+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad74142520 con 0x7fad7413f710 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957559723 0 0) 0x7fad740b28a0 con 0x7fad741430c0 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 10 monclient(hunting): my global_id is 4633 2024-03-25T19:51:03.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.555+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad740b2a30 con 0x7fad741430c0 2024-03-25T19:51:03.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659537132 0 0) 0x7fad740b2a30 con 0x7fad741430c0 2024-03-25T19:51:03.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 >> v1:172.21.15.130:6790/0 conn(0x7fad7413f710 legacy=0x7fad741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 >> v1:172.21.15.130:6789/0 conn(0x7fad740a98f0 legacy=0x7fad740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad740b44c0 con 0x7fad741430c0 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad6c001930 con 0x7fad741430c0 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad737fe640 10 monclient: dump: 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:03.571 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad7a590640 5 monclient: authenticate success, global_id 4633 2024-03-25T19:51:03.572 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.569+0000 7fad7a590640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:03.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad6c001a90 con 0x7fad741430c0 2024-03-25T19:51:03.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:03.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 1 -- 172.21.15.130:0/1824519520 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad6c001ee0 con 0x7fad741430c0 2024-03-25T19:51:03.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad7a590640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:03.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad7a590640 4 set_mon_vals no callback set 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.581+0000 7fad737fe640 10 monclient: dump: 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:03.583 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 10 monclient: shutdown 2024-03-25T19:51:03.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:03.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 1 -- 172.21.15.130:0/1824519520 >> v1:172.21.15.163:6789/0 conn(0x7fad741430c0 legacy=0x7fad7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 1 -- 172.21.15.130:0/1824519520 shutdown_connections 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 1 -- 172.21.15.130:0/1824519520 >> 172.21.15.130:0/1824519520 conn(0x7fad740b7ea0 msgr2=0x7fad740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 1 -- 172.21.15.130:0/1824519520 shutdown_connections 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.582+0000 7fad7a590640 1 -- 172.21.15.130:0/1824519520 wait complete. 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.583+0000 7fad7a590640 10 monclient: build_initial_monmap 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.583+0000 7fad7a590640 10 monclient: monmap: 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:03.586 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 librados: starting msgr at 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 librados: starting objecter 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 Processor -- start 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 -- start start 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 librados: setting wanted keys 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 1 librados: calling monclient init 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient: init 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-c con 0x7fad740a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-a con 0x7fad7413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.584+0000 7fad7a590640 10 monclient(hunting): picked mon.noname-b con 0x7fad741430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.585+0000 7fad7a590640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad741397e0 con 0x7fad7413f710 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.585+0000 7fad7a590640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740ab9e0 con 0x7fad741430c0 2024-03-25T19:51:03.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.585+0000 7fad7a590640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad740abcc0 con 0x7fad740a98f0 2024-03-25T19:51:03.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.585+0000 7fad7a590640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:03.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.585+0000 7fad7a590640 10 monclient(hunting): authenticate will time out at 5539.731709s 2024-03-25T19:51:03.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.599+0000 7fad6bfff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fad7413f710 0x7fad740ac150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:42550/0 (socket says 172.21.15.130:42550) 2024-03-25T19:51:03.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.599+0000 7fad6bfff640 1 -- 172.21.15.130:0/964367973 learned_addr learned my addr 172.21.15.130:0/964367973 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:03.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.601+0000 7fad717fa640 10 client.?.objecter ms_handle_connect 0x7fad7413f710 2024-03-25T19:51:03.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.601+0000 7fad717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.605+0000 7fad717fa640 10 client.?.objecter ms_handle_connect 0x7fad741430c0 2024-03-25T19:51:03.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.605+0000 7fad717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.617 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.617+0000 7fad717fa640 10 client.?.objecter ms_handle_connect 0x7fad740a98f0 2024-03-25T19:51:03.617 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.617+0000 7fad717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:03.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 776646829 0 0) 0x7fad740abcc0 con 0x7fad740a98f0 2024-03-25T19:51:03.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 10 monclient(hunting): my global_id is 4634 2024-03-25T19:51:03.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.625+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad54004eb0 con 0x7fad740a98f0 2024-03-25T19:51:03.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591444521 0 0) 0x7fad741397e0 con 0x7fad7413f710 2024-03-25T19:51:03.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 10 monclient(hunting): my global_id is 4641 2024-03-25T19:51:03.630 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.629+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad740abcc0 con 0x7fad7413f710 2024-03-25T19:51:03.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1992313958 0 0) 0x7fad740ab9e0 con 0x7fad741430c0 2024-03-25T19:51:03.632 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:03.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:03.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:03.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 10 monclient(hunting): my global_id is 4636 2024-03-25T19:51:03.633 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.631+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad741397e0 con 0x7fad741430c0 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3884644225 0 0) 0x7fad741397e0 con 0x7fad741430c0 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad740ab9e0 con 0x7fad741430c0 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad6c001da0 con 0x7fad741430c0 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 10 monclient(hunting): dump: 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:03.652 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:03.653 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:03.653 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.653 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.651+0000 7fad717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:03.653 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619304072 0 0) 0x7fad54004eb0 con 0x7fad740a98f0 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad741397e0 con 0x7fad740a98f0 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad64001950 con 0x7fad740a98f0 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.653+0000 7fad717fa640 10 monclient(hunting): dump: 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:03.654 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:03.655 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2358023428 0 0) 0x7fad740abcc0 con 0x7fad7413f710 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad54004eb0 con 0x7fad7413f710 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad5c0019f0 con 0x7fad7413f710 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.657+0000 7fad717fa640 10 monclient(hunting): dump: 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:03.658 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:03.659 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:03.659 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3988900855 0 0) 0x7fad54004eb0 con 0x7fad7413f710 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 >> v1:172.21.15.130:6790/0 conn(0x7fad740a98f0 legacy=0x7fad741384f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 >> v1:172.21.15.163:6789/0 conn(0x7fad741430c0 legacy=0x7fad741e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 1 monclient: found mon.a 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:03.671 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad74138d30 con 0x7fad7413f710 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 5 monclient: authenticate success, global_id 4641 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 10 monclient: _renew_subs 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad74137970 con 0x7fad7413f710 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 10 client.4641.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 10 monclient: _renew_subs 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad74136420 con 0x7fad7413f710 2024-03-25T19:51:03.672 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.670+0000 7fad7a590640 1 librados: init done 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.671+0000 7fad7a590640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.671+0000 7fad7a590640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.671+0000 7fad7a590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:03.673 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:03.671+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad740aa5a0 con 0x7fad7413f710 2024-03-25T19:51:04.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad5c003610 con 0x7fad7413f710 2024-03-25T19:51:04.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:04.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fad5c003a20 con 0x7fad7413f710 2024-03-25T19:51:04.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad717fa640 10 monclient: dump: 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:04.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.003+0000 7fad72ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:04.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.004+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fad5c003c40 con 0x7fad7413f710 2024-03-25T19:51:04.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.005+0000 7fad73fff640 10 monclient: get_auth_request con 0x7fad5406e160 auth_method 0 2024-03-25T19:51:04.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.005+0000 7fad717fa640 10 client.4641.objecter ms_handle_connect 0x7fad5406e160 2024-03-25T19:51:04.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.010+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fad5c0547e0 con 0x7fad7413f710 2024-03-25T19:51:04.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.010+0000 7fad717fa640 10 client.4641.objecter ms_dispatch 0x7fad74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:04.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.010+0000 7fad717fa640 3 client.4641.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:04.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.010+0000 7fad717fa640 3 client.4641.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:04.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.010+0000 7fad717fa640 20 client.4641.objecter dump_active .. 0 homeless 2024-03-25T19:51:04.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.011+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad5c0502f0 con 0x7fad7413f710 2024-03-25T19:51:04.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.011+0000 7fad717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:04.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.011+0000 7fad717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:04.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.245+0000 7fad7a590640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.245+0000 7fad7a590640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.245+0000 7fad7a590640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:04.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.245+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad74138ec0 con 0x7fad7413f710 2024-03-25T19:51:04.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.353+0000 7fad717fa640 1 -- 172.21.15.130:0/964367973 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fad5c052220 con 0x7fad7413f710 2024-03-25T19:51:04.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.353+0000 7fad717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:04.354 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.353+0000 7fad717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:04.355 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:04.355 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 10 librados: watch_flush enter 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 10 librados: watch_flush exit 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 20 client.4641.objecter shutdown clearing up homeless session... 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 10 client.4641.objecter successfully canceled tick 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 >> v1:172.21.15.163:6800/1217380380 conn(0x7fad5406e160 legacy=0x7fad54070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.357 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 10 monclient: shutdown 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 >> v1:172.21.15.130:6789/0 conn(0x7fad7413f710 legacy=0x7fad740ac150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 shutdown_connections 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 >> 172.21.15.130:0/964367973 conn(0x7fad740b7ea0 msgr2=0x7fad74074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 shutdown_connections 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 -- 172.21.15.130:0/964367973 wait complete. 2024-03-25T19:51:04.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:04.356+0000 7fad7a590640 1 librados: shutdown 2024-03-25T19:51:04.369 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:10.645 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:55.578 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.654 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:55.587 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.698 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:55.599 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.702 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:09:55.635 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.712 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:55.645 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.714 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:55.615 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.727 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:55.661 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.739 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:09:55.640 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:10.761 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:09:55.695 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:11.176 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.314+0000 7f5927f7c640 10 monclient: get_monmap_and_config 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.314+0000 7f5927f7c640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.314+0000 7f5927f7c640 10 monclient: monmap: 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:11.315610+0000 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:11.315610+0000 2024-03-25T19:51:11.314 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.315 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:11.315 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:11.315 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:11.315 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:11.315 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.315+0000 7f5927f7c640 1 Processor -- start 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.315+0000 7f5927f7c640 1 -- start start 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.315+0000 7f5927f7c640 10 monclient: init 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f59200b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:11.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f59200b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f592014be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200af940 con 0x7f59200b1730 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200b0a70 con 0x7f592014be90 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59200b0d50 con 0x7f59200b4d60 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.316+0000 7f5927f7c640 10 monclient(hunting): authenticate will time out at 5547.462368s 2024-03-25T19:51:11.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.324+0000 7f5926519640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f592014be90 0x7f592014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:33384/0 (socket says 172.21.15.130:33384) 2024-03-25T19:51:11.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.324+0000 7f5926519640 1 -- 172.21.15.130:0/2026380713 learned_addr learned my addr 172.21.15.130:0/2026380713 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169230950 0 0) 0x7f59200af940 con 0x7f59200b1730 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 10 monclient(hunting): my global_id is 4644 2024-03-25T19:51:11.347 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.347+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f590c003520 con 0x7f59200b1730 2024-03-25T19:51:11.358 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2439209299 0 0) 0x7f59200b0a70 con 0x7f592014be90 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient(hunting): my global_id is 4639 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59200af940 con 0x7f592014be90 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 263142210 0 0) 0x7f590c003520 con 0x7f59200b1730 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 >> v1:172.21.15.130:6790/0 conn(0x7f59200b4d60 legacy=0x7f592013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 >> v1:172.21.15.163:6789/0 conn(0x7f592014be90 legacy=0x7f592014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 monclient: found mon.noname-a 2024-03-25T19:51:11.359 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59200b1030 con 0x7f59200b1730 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5910001690 con 0x7f59200b1730 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5924d16640 10 monclient: dump: 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.360 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5927f7c640 5 monclient: authenticate success, global_id 4644 2024-03-25T19:51:11.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.358+0000 7f5927f7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:11.378 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5910001870 con 0x7f59200b1730 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 1 -- 172.21.15.130:0/2026380713 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5910001690 con 0x7f59200b1730 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 4 set_mon_vals no callback set 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5924d16640 10 monclient: dump: 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:11.379 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:11.380 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:11.380 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:11.380 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 10 monclient: shutdown 2024-03-25T19:51:11.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 1 -- 172.21.15.130:0/2026380713 >> v1:172.21.15.130:6789/0 conn(0x7f59200b1730 legacy=0x7f59200b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 1 -- 172.21.15.130:0/2026380713 shutdown_connections 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.379+0000 7f5927f7c640 1 -- 172.21.15.130:0/2026380713 >> 172.21.15.130:0/2026380713 conn(0x7f59200617a0 msgr2=0x7f5920061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.380+0000 7f5927f7c640 1 -- 172.21.15.130:0/2026380713 shutdown_connections 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.380+0000 7f5927f7c640 1 -- 172.21.15.130:0/2026380713 wait complete. 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.380+0000 7f5927f7c640 10 monclient: build_initial_monmap 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.380+0000 7f5927f7c640 10 monclient: monmap: 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:11.382 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 librados: starting msgr at 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 librados: starting objecter 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 Processor -- start 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 -- start start 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 librados: setting wanted keys 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 1 librados: calling monclient init 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 10 monclient: init 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:11.383 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.381+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-a con 0x7f59200b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-b con 0x7f59200b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 10 monclient(hunting): picked mon.noname-c con 0x7f592014be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5920134a00 con 0x7f59200b1730 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59201423b0 con 0x7f59200b4d60 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5920140e30 con 0x7f592014be90 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:11.384 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.382+0000 7f5927f7c640 10 monclient(hunting): authenticate will time out at 5547.527723s 2024-03-25T19:51:11.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.390+0000 7f5925517640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f59200b4d60 0x7f5920133be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:33436/0 (socket says 172.21.15.130:33436) 2024-03-25T19:51:11.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.390+0000 7f5925517640 1 -- 172.21.15.130:0/4252260149 learned_addr learned my addr 172.21.15.130:0/4252260149 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:11.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.390+0000 7f59167fc640 10 client.?.objecter ms_handle_connect 0x7f59200b4d60 2024-03-25T19:51:11.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.390+0000 7f59167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.393+0000 7f59167fc640 10 client.?.objecter ms_handle_connect 0x7f59200b1730 2024-03-25T19:51:11.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.393+0000 7f59167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640240672 0 0) 0x7f59201423b0 con 0x7f59200b4d60 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 monclient(hunting): my global_id is 4642 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5904004eb0 con 0x7f59200b4d60 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 client.?.objecter ms_handle_connect 0x7f592014be90 2024-03-25T19:51:11.402 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.402+0000 7f59167fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:11.406 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900961189 0 0) 0x7f5920134a00 con 0x7f59200b1730 2024-03-25T19:51:11.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T19:51:11.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.407+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59201423b0 con 0x7f59200b1730 2024-03-25T19:51:11.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.408+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1449794467 0 0) 0x7f5904004eb0 con 0x7f59200b4d60 2024-03-25T19:51:11.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5920134a00 con 0x7f59200b4d60 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f59100035f0 con 0x7f59200b4d60 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 10 monclient(hunting): dump: 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:11.409 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:11.410 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:11.410 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.410 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.409+0000 7f59167fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 914497892 0 0) 0x7f5920140e30 con 0x7f592014be90 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 10 monclient(hunting): _negotiate 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T19:51:11.418 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.418+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5904004eb0 con 0x7f592014be90 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3150156856 0 0) 0x7f5920134a00 con 0x7f59200b4d60 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 >> v1:172.21.15.130:6790/0 conn(0x7f592014be90 legacy=0x7f59201342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 >> v1:172.21.15.130:6789/0 conn(0x7f59200b1730 legacy=0x7f5920140720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 10 monclient: _finish_hunting 0 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 1 monclient: found mon.b 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59201410b0 con 0x7f59200b4d60 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 10 monclient: _finish_auth 0 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 10 monclient: _check_auth_tickets 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f59167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 5 monclient: authenticate success, global_id 4642 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 10 monclient: _renew_subs 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59201412a0 con 0x7f59200b4d60 2024-03-25T19:51:11.427 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:11.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 10 monclient: _renew_subs 2024-03-25T19:51:11.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:11.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5920137c40 con 0x7f59200b4d60 2024-03-25T19:51:11.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.426+0000 7f5927f7c640 1 librados: init done 2024-03-25T19:51:11.428 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.428+0000 7f5927f7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.428+0000 7f5927f7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.428+0000 7f5927f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:11.429 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.428+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59201421b0 con 0x7f59200b4d60 2024-03-25T19:51:11.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.707+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5910003a00 con 0x7f59200b4d60 2024-03-25T19:51:11.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.707+0000 7f59167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:11.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f5910003e10 con 0x7f59200b4d60 2024-03-25T19:51:11.707 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f59167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f59167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f59167fc640 10 monclient: dump: 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:11.708 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f5917fff640 4 set_mon_vals no callback set 2024-03-25T19:51:11.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.708+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f5910004030 con 0x7f59200b4d60 2024-03-25T19:51:11.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.709+0000 7f5925d18640 10 monclient: get_auth_request con 0x7f590406de00 auth_method 0 2024-03-25T19:51:11.709 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.709+0000 7f59167fc640 10 client.4642.objecter ms_handle_connect 0x7f590406de00 2024-03-25T19:51:11.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f59100011e0 con 0x7f59200b4d60 2024-03-25T19:51:11.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 10 client.4642.objecter ms_dispatch 0x7f5920000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:11.714 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:11.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:11.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T19:51:11.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5910050500 con 0x7f59200b4d60 2024-03-25T19:51:11.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:11.715 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.715+0000 7f59167fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:11.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.960+0000 7f5927f7c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:11.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.960+0000 7f5927f7c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:11.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.960+0000 7f5927f7c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:11.960 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.960+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59200627b0 con 0x7f59200b4d60 2024-03-25T19:51:11.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.990+0000 7f59167fc640 1 -- 172.21.15.130:0/4252260149 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f5910052430 con 0x7f59200b4d60 2024-03-25T19:51:11.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.990+0000 7f59167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:11.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.990+0000 7f59167fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:11.990 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:11.991 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 10 librados: watch_flush enter 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 10 librados: watch_flush exit 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 10 client.4642.objecter successfully canceled tick 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 >> v1:172.21.15.163:6800/1217380380 conn(0x7f590406de00 legacy=0x7f59040702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 10 monclient: shutdown 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 >> v1:172.21.15.163:6789/0 conn(0x7f59200b4d60 legacy=0x7f5920133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 shutdown_connections 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 >> 172.21.15.130:0/4252260149 conn(0x7f59200617a0 msgr2=0x7f592013c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 shutdown_connections 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 -- 172.21.15.130:0/4252260149 wait complete. 2024-03-25T19:51:11.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:11.993+0000 7f5927f7c640 1 librados: shutdown 2024-03-25T19:51:12.005 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:18.811 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:18.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.931+0000 7f3c11255640 10 monclient: get_monmap_and_config 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.931+0000 7f3c11255640 10 monclient: build_initial_monmap 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.931+0000 7f3c11255640 10 monclient: monmap: 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:18.931913+0000 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:18.931913+0000 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:18.931 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:18.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.931+0000 7f3c11255640 1 Processor -- start 2024-03-25T19:51:18.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 1 -- start start 2024-03-25T19:51:18.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient: init 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c142520 con 0x7f3c0c13f710 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c0b28a0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c0b2a30 con 0x7f3c0c1430c0 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:18.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.932+0000 7f3c11255640 10 monclient(hunting): authenticate will time out at 5555.078525s 2024-03-25T19:51:18.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.934+0000 7f3c0ad9d640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3c0c0a98f0 0x7f3c0c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:42298/0 (socket says 172.21.15.130:42298) 2024-03-25T19:51:18.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.934+0000 7f3c0ad9d640 1 -- 172.21.15.130:0/69471464 learned_addr learned my addr 172.21.15.130:0/69471464 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:18.941 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335169327 0 0) 0x7f3c0c0b28a0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 10 monclient(hunting): _negotiate 2024-03-25T19:51:18.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:18.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:18.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 10 monclient(hunting): my global_id is 4645 2024-03-25T19:51:18.942 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.942+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bf8003520 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032267265 0 0) 0x7f3c0c0b2a30 con 0x7f3c0c1430c0 2024-03-25T19:51:18.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient(hunting): _negotiate 2024-03-25T19:51:18.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient(hunting): my global_id is 4643 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c0b28a0 con 0x7f3c0c1430c0 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3866822668 0 0) 0x7f3bf8003520 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 >> v1:172.21.15.130:6790/0 conn(0x7f3c0c1430c0 legacy=0x7f3c0c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 >> v1:172.21.15.130:6789/0 conn(0x7f3c0c13f710 legacy=0x7f3c0c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: _finish_hunting 0 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 monclient: found mon.noname-c 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c0b44c0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: _finish_auth 0 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: _check_auth_tickets 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bfc001690 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c0a59c640 10 monclient: dump: 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:18.955 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:18.956 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:18.956 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:18.956 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:18.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c11255640 5 monclient: authenticate success, global_id 4645 2024-03-25T19:51:18.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.953+0000 7f3c11255640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bfc0017f0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 1 -- 172.21.15.130:0/69471464 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bfc001c00 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c0a59c640 10 monclient: dump: 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:18.964 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c11255640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:18.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.964+0000 7f3c11255640 4 set_mon_vals no callback set 2024-03-25T19:51:18.966 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 10 monclient: shutdown 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 1 -- 172.21.15.130:0/69471464 >> v1:172.21.15.163:6789/0 conn(0x7f3c0c0a98f0 legacy=0x7f3c0c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 1 -- 172.21.15.130:0/69471464 shutdown_connections 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 1 -- 172.21.15.130:0/69471464 >> 172.21.15.130:0/69471464 conn(0x7f3c0c0b7ea0 msgr2=0x7f3c0c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 1 -- 172.21.15.130:0/69471464 shutdown_connections 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.965+0000 7f3c11255640 1 -- 172.21.15.130:0/69471464 wait complete. 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.966+0000 7f3c11255640 10 monclient: build_initial_monmap 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.966+0000 7f3c11255640 10 monclient: monmap: 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:18.967 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 librados: starting msgr at 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 librados: starting objecter 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 Processor -- start 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 -- start start 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 librados: setting wanted keys 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 librados: calling monclient init 2024-03-25T19:51:18.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient: init 2024-03-25T19:51:18.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:18.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:18.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-a con 0x7f3c0c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:18.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-b con 0x7f3c0c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient(hunting): picked mon.noname-c con 0x7f3c0c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c132ba0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c138150 con 0x7f3c0c13f710 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0c139510 con 0x7f3c0c1430c0 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:18.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.967+0000 7f3c11255640 10 monclient(hunting): authenticate will time out at 5555.113495s 2024-03-25T19:51:18.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.973+0000 7f3c03fff640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f3c0c13f710 0x7f3c0c135000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:42302/0 (socket says 172.21.15.130:42302) 2024-03-25T19:51:18.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.973+0000 7f3c03fff640 1 -- 172.21.15.130:0/2493344576 learned_addr learned my addr 172.21.15.130:0/2493344576 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:18.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.974+0000 7f3c037fe640 10 client.?.objecter ms_handle_connect 0x7f3c0c13f710 2024-03-25T19:51:18.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.974+0000 7f3c037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:18.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.978+0000 7f3c037fe640 10 client.?.objecter ms_handle_connect 0x7f3c0c0a98f0 2024-03-25T19:51:18.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.978+0000 7f3c037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:18.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.981+0000 7f3c037fe640 10 client.?.objecter ms_handle_connect 0x7f3c0c1430c0 2024-03-25T19:51:18.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.981+0000 7f3c037fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:18.983 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 614546872 0 0) 0x7f3c0c138150 con 0x7f3c0c13f710 2024-03-25T19:51:18.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:18.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:18.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:18.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T19:51:18.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.984+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec004eb0 con 0x7f3c0c13f710 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1792882993 0 0) 0x7f3bec004eb0 con 0x7f3c0c13f710 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c0c138150 con 0x7f3c0c13f710 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bf4001a30 con 0x7f3c0c13f710 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): dump: 2024-03-25T19:51:18.994 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2741045929 0 0) 0x7f3c0c0b44c0 con 0x7f3c0c0a98f0 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:18.995 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:18.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T19:51:18.996 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:18.994+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec004eb0 con 0x7f3c0c0a98f0 2024-03-25T19:51:19.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340574353 0 0) 0x7f3c0c139510 con 0x7f3c0c1430c0 2024-03-25T19:51:19.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:19.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:19.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:19.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T19:51:19.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.006+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c0c0b44c0 con 0x7f3c0c1430c0 2024-03-25T19:51:19.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1209737682 0 0) 0x7f3bec004eb0 con 0x7f3c0c0a98f0 2024-03-25T19:51:19.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c0c139510 con 0x7f3c0c0a98f0 2024-03-25T19:51:19.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bfc001960 con 0x7f3c0c0a98f0 2024-03-25T19:51:19.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.008 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.008+0000 7f3c037fe640 10 monclient(hunting): dump: 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:19.009 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:19.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 776286870 0 0) 0x7f3c0c138150 con 0x7f3c0c13f710 2024-03-25T19:51:19.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 >> v1:172.21.15.130:6790/0 conn(0x7f3c0c1430c0 legacy=0x7f3c0c1e27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 >> v1:172.21.15.130:6789/0 conn(0x7f3c0c0a98f0 legacy=0x7f3c0c137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.010 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 1 monclient: found mon.b 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c1397f0 con 0x7f3c0c13f710 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 5 monclient: authenticate success, global_id 4648 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 10 monclient: _renew_subs 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c138600 con 0x7f3c0c13f710 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 10 monclient: _renew_subs 2024-03-25T19:51:19.011 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0c135950 con 0x7f3c0c13f710 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.010+0000 7f3c11255640 1 librados: init done 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.011+0000 7f3c11255640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.011+0000 7f3c11255640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.011+0000 7f3c11255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:19.012 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.011+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0c138320 con 0x7f3c0c13f710 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3bf4001cb0 con 0x7f3c0c13f710 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f3bf40038c0 con 0x7f3c0c13f710 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c037fe640 10 monclient: dump: 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:19.279 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:19.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.279+0000 7f3c09d9b640 4 set_mon_vals no callback set 2024-03-25T19:51:19.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.280+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f3bf4003aa0 con 0x7f3c0c13f710 2024-03-25T19:51:19.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.281+0000 7f3c0ad9d640 10 monclient: get_auth_request con 0x7f3bec06e040 auth_method 0 2024-03-25T19:51:19.281 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.281+0000 7f3c037fe640 10 client.4648.objecter ms_handle_connect 0x7f3bec06e040 2024-03-25T19:51:19.284 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.285+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f3bf4054710 con 0x7f3c0c13f710 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.285+0000 7f3c037fe640 10 client.4648.objecter ms_dispatch 0x7f3c0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.285+0000 7f3c037fe640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.285+0000 7f3c037fe640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.285+0000 7f3c037fe640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.286+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3bf4050220 con 0x7f3c0c13f710 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.286+0000 7f3c037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:19.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.286+0000 7f3c037fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.520+0000 7f3c11255640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.520+0000 7f3c11255640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.520+0000 7f3c11255640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:19.520 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.520+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c0c0627b0 con 0x7f3c0c13f710 2024-03-25T19:51:19.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.548+0000 7f3c037fe640 1 -- 172.21.15.130:0/2493344576 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f3bf4052150 con 0x7f3c0c13f710 2024-03-25T19:51:19.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.548+0000 7f3c037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:19.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.548+0000 7f3c037fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:19.548 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:19.548 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 10 librados: watch_flush enter 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 10 librados: watch_flush exit 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 10 client.4648.objecter successfully canceled tick 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 >> v1:172.21.15.163:6800/1217380380 conn(0x7f3bec06e040 legacy=0x7f3bec070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 10 monclient: shutdown 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.550+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 >> v1:172.21.15.163:6789/0 conn(0x7f3c0c13f710 legacy=0x7f3c0c135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.551+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 shutdown_connections 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.551+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 >> 172.21.15.130:0/2493344576 conn(0x7f3c0c0b7ea0 msgr2=0x7f3c0c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.551+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 shutdown_connections 2024-03-25T19:51:19.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.551+0000 7f3c11255640 1 -- 172.21.15.130:0/2493344576 wait complete. 2024-03-25T19:51:19.552 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:19.551+0000 7f3c11255640 1 librados: shutdown 2024-03-25T19:51:19.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:26.369 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.487+0000 7f6f9363e640 10 monclient: get_monmap_and_config 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.487+0000 7f6f9363e640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.487+0000 7f6f9363e640 10 monclient: monmap: 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:26.488271+0000 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:26.488271+0000 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.487 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:26.488 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:26.488 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:26.488 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 1 Processor -- start 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 1 -- start start 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient: init 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8c0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c142520 con 0x7f6f8c13f710 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c0b28a0 con 0x7f6f8c1430c0 2024-03-25T19:51:26.489 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c0b2a30 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.490 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.488+0000 7f6f9363e640 10 monclient(hunting): authenticate will time out at 5562.634535s 2024-03-25T19:51:26.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.496+0000 7f6f91bdb640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f6f8c1430c0 0x7f6f8c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:46874/0 (socket says 172.21.15.130:46874) 2024-03-25T19:51:26.496 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.496+0000 7f6f91bdb640 1 -- 172.21.15.130:0/2255227070 learned_addr learned my addr 172.21.15.130:0/2255227070 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4138093521 0 0) 0x7f6f8c0b2a30 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 10 monclient(hunting): my global_id is 4649 2024-03-25T19:51:26.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.507+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f74003520 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3637696902 0 0) 0x7f6f8c0b28a0 con 0x7f6f8c1430c0 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 10 monclient(hunting): my global_id is 4651 2024-03-25T19:51:26.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.508+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f8c0b2a30 con 0x7f6f8c1430c0 2024-03-25T19:51:26.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1460999434 0 0) 0x7f6f74003520 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 >> v1:172.21.15.163:6789/0 conn(0x7f6f8c1430c0 legacy=0x7f6f8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 >> v1:172.21.15.130:6789/0 conn(0x7f6f8c13f710 legacy=0x7f6f8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.516 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 monclient: found mon.noname-b 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c0b44c0 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6f7c001a10 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f9363e640 5 monclient: authenticate success, global_id 4649 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f9363e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f8bfff640 10 monclient: dump: 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.517 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.518 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.516+0000 7f6f9363e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f7c001b70 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 1 -- 172.21.15.130:0/2255227070 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6f7c001f80 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f9363e640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f9363e640 4 set_mon_vals no callback set 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.527+0000 7f6f8bfff640 10 monclient: dump: 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.527 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:26.528 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 10 monclient: shutdown 2024-03-25T19:51:26.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:26.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 1 -- 172.21.15.130:0/2255227070 >> v1:172.21.15.130:6790/0 conn(0x7f6f8c0a98f0 legacy=0x7f6f8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 1 -- 172.21.15.130:0/2255227070 shutdown_connections 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 1 -- 172.21.15.130:0/2255227070 >> 172.21.15.130:0/2255227070 conn(0x7f6f8c0b7ea0 msgr2=0x7f6f8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 1 -- 172.21.15.130:0/2255227070 shutdown_connections 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.528+0000 7f6f9363e640 1 -- 172.21.15.130:0/2255227070 wait complete. 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.529+0000 7f6f9363e640 10 monclient: build_initial_monmap 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.529+0000 7f6f9363e640 10 monclient: monmap: 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:26.530 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.529+0000 7f6f9363e640 1 librados: starting msgr at 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.529+0000 7f6f9363e640 1 librados: starting objecter 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 1 Processor -- start 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 1 -- start start 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 1 librados: setting wanted keys 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 1 librados: calling monclient init 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 10 monclient: init 2024-03-25T19:51:26.531 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.530+0000 7f6f9363e640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-b con 0x7f6f8c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-a con 0x7f6f8c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient(hunting): picked mon.noname-c con 0x7f6f8c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c0ab9e0 con 0x7f6f8c13f710 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c138c20 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f8c138f00 con 0x7f6f8c1430c0 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:26.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.531+0000 7f6f9363e640 10 monclient(hunting): authenticate will time out at 5562.676865s 2024-03-25T19:51:26.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.541+0000 7f6f90bd9640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f6f8c13f710 0x7f6f8c1d7950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56152/0 (socket says 172.21.15.130:56152) 2024-03-25T19:51:26.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.541+0000 7f6f90bd9640 1 -- 172.21.15.130:0/3400529167 learned_addr learned my addr 172.21.15.130:0/3400529167 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:26.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.542+0000 7f6f89ffb640 10 client.?.objecter ms_handle_connect 0x7f6f8c13f710 2024-03-25T19:51:26.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.542+0000 7f6f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.548+0000 7f6f89ffb640 10 client.?.objecter ms_handle_connect 0x7f6f8c0a98f0 2024-03-25T19:51:26.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.548+0000 7f6f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.550+0000 7f6f89ffb640 10 client.?.objecter ms_handle_connect 0x7f6f8c1430c0 2024-03-25T19:51:26.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.550+0000 7f6f89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718136276 0 0) 0x7f6f8c0ab9e0 con 0x7f6f8c13f710 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 10 monclient(hunting): my global_id is 4659 2024-03-25T19:51:26.551 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.551+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f78004eb0 con 0x7f6f8c13f710 2024-03-25T19:51:26.567 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503256481 0 0) 0x7f6f8c138c20 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 10 monclient(hunting): my global_id is 4654 2024-03-25T19:51:26.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.568+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f8c0ab9e0 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058492840 0 0) 0x7f6f8c138f00 con 0x7f6f8c1430c0 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 10 monclient(hunting): my global_id is 4652 2024-03-25T19:51:26.570 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.570+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f8c138c20 con 0x7f6f8c1430c0 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3273875972 0 0) 0x7f6f78004eb0 con 0x7f6f8c13f710 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f8c138f00 con 0x7f6f8c13f710 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6f800019d0 con 0x7f6f8c13f710 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 10 monclient(hunting): dump: 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:26.583 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:26.584 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.583+0000 7f6f89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3098058027 0 0) 0x7f6f8c0ab9e0 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f78004eb0 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6f7c001e80 con 0x7f6f8c0a98f0 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.595+0000 7f6f89ffb640 10 monclient(hunting): dump: 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.595 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:26.596 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2705006123 0 0) 0x7f6f8c138f00 con 0x7f6f8c13f710 2024-03-25T19:51:26.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 >> v1:172.21.15.130:6790/0 conn(0x7f6f8c1430c0 legacy=0x7f6f8c1e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 >> v1:172.21.15.163:6789/0 conn(0x7f6f8c0a98f0 legacy=0x7f6f8c137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:26.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 1 monclient: found mon.a 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f8c14e7a0 con 0x7f6f8c13f710 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 5 monclient: authenticate success, global_id 4659 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 10 monclient: _renew_subs 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c1386d0 con 0x7f6f8c13f710 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 10 monclient: _renew_subs 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:26.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c1d8060 con 0x7f6f8c13f710 2024-03-25T19:51:26.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.598+0000 7f6f9363e640 1 librados: init done 2024-03-25T19:51:26.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.599+0000 7f6f9363e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.599+0000 7f6f9363e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.599+0000 7f6f9363e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:26.600 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.599+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c0627b0 con 0x7f6f8c13f710 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f80003610 con 0x7f6f8c13f710 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f6f80003a20 con 0x7f6f8c13f710 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f89ffb640 10 monclient: dump: 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:26.901 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.901+0000 7f6f8b7fe640 4 set_mon_vals no callback set 2024-03-25T19:51:26.902 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.902+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f6f80003c40 con 0x7f6f8c13f710 2024-03-25T19:51:26.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.902+0000 7f6f913da640 10 monclient: get_auth_request con 0x7f6f7806df10 auth_method 0 2024-03-25T19:51:26.903 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.903+0000 7f6f89ffb640 10 client.4659.objecter ms_handle_connect 0x7f6f7806df10 2024-03-25T19:51:26.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7f6f800547f0 con 0x7f6f8c13f710 2024-03-25T19:51:26.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 10 client.4659.objecter ms_dispatch 0x7f6f8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:26.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:26.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:26.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T19:51:26.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f80050300 con 0x7f6f8c13f710 2024-03-25T19:51:26.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:26.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:26.909+0000 7f6f89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:27.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.156+0000 7f6f9363e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.156+0000 7f6f9363e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.156+0000 7f6f9363e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:51:27.156 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.156+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f8c074cc0 con 0x7f6f8c13f710 2024-03-25T19:51:27.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.210+0000 7f6f89ffb640 1 -- 172.21.15.130:0/3400529167 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7f6f80052230 con 0x7f6f8c13f710 2024-03-25T19:51:27.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.210+0000 7f6f89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:27.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.210+0000 7f6f89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:27.210 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:27.210 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 10 librados: watch_flush enter 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 10 librados: watch_flush exit 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 10 client.4659.objecter successfully canceled tick 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 >> v1:172.21.15.163:6800/1217380380 conn(0x7f6f7806df10 legacy=0x7f6f780703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 10 monclient: shutdown 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 >> v1:172.21.15.130:6789/0 conn(0x7f6f8c13f710 legacy=0x7f6f8c1d7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 shutdown_connections 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 >> 172.21.15.130:0/3400529167 conn(0x7f6f8c0b7ea0 msgr2=0x7f6f8c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:27.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 shutdown_connections 2024-03-25T19:51:27.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 -- 172.21.15.130:0/3400529167 wait complete. 2024-03-25T19:51:27.214 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:27.213+0000 7f6f9363e640 1 librados: shutdown 2024-03-25T19:51:27.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:30.658 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:15.591 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.710 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:15.643 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.743 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:15.676 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.759 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:15.660 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.760 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:15.693 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.769 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:15.670 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.774 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:15.707 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.798 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:15.699 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:30.801 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:15.735 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:34.032 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.147+0000 7fd94d197640 10 monclient: get_monmap_and_config 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.147+0000 7fd94d197640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.147+0000 7fd94d197640 10 monclient: monmap: 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.147 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:34.148688+0000 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:34.148688+0000 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:34.148 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.148+0000 7fd94d197640 1 Processor -- start 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.148+0000 7fd94d197640 1 -- start start 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.148+0000 7fd94d197640 10 monclient: init 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-c con 0x7fd9480b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-a con 0x7fd9480b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:34.149 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-b con 0x7fd94814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:34.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9480af940 con 0x7fd9480b1730 2024-03-25T19:51:34.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9480b0a70 con 0x7fd9480b4d60 2024-03-25T19:51:34.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9480b0d50 con 0x7fd94814be90 2024-03-25T19:51:34.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.150 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.149+0000 7fd94d197640 10 monclient(hunting): authenticate will time out at 5570.295323s 2024-03-25T19:51:34.153 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.154+0000 7fd94659c640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fd9480b1730 0x7fd9480b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:44082/0 (socket says 172.21.15.130:44082) 2024-03-25T19:51:34.153 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.154+0000 7fd94659c640 1 -- 172.21.15.130:0/488480909 learned_addr learned my addr 172.21.15.130:0/488480909 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802612363 0 0) 0x7fd9480af940 con 0x7fd9480b1730 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 10 monclient(hunting): my global_id is 4662 2024-03-25T19:51:34.162 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.162+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd930003520 con 0x7fd9480b1730 2024-03-25T19:51:34.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3459765338 0 0) 0x7fd930003520 con 0x7fd9480b1730 2024-03-25T19:51:34.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 >> v1:172.21.15.130:6790/0 conn(0x7fd94814be90 legacy=0x7fd94814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 >> v1:172.21.15.163:6789/0 conn(0x7fd9480b4d60 legacy=0x7fd94813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.176 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 1 monclient: found mon.noname-a 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.176+0000 7fd945d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9480b1030 con 0x7fd9480b1730 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd93c001690 con 0x7fd9480b1730 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd94d197640 5 monclient: authenticate success, global_id 4662 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd94d197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd945d9b640 10 monclient: dump: 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.177 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.178 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.177+0000 7fd94d197640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd93c0017f0 con 0x7fd9480b1730 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd94d197640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd94d197640 4 set_mon_vals no callback set 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 1 -- 172.21.15.130:0/488480909 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd93c001c00 con 0x7fd9480b1730 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.179+0000 7fd945d9b640 10 monclient: dump: 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.179 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.180 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.180 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:34.180 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:34.180 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:34.180 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 10 monclient: shutdown 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 1 -- 172.21.15.130:0/488480909 >> v1:172.21.15.130:6789/0 conn(0x7fd9480b1730 legacy=0x7fd9480b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 1 -- 172.21.15.130:0/488480909 shutdown_connections 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 1 -- 172.21.15.130:0/488480909 >> 172.21.15.130:0/488480909 conn(0x7fd9480617a0 msgr2=0x7fd948061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 1 -- 172.21.15.130:0/488480909 shutdown_connections 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.180+0000 7fd94d197640 1 -- 172.21.15.130:0/488480909 wait complete. 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 10 monclient: build_initial_monmap 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 10 monclient: monmap: 2024-03-25T19:51:34.181 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 librados: starting msgr at 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 librados: starting objecter 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 Processor -- start 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 -- start start 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 librados: setting wanted keys 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 1 librados: calling monclient init 2024-03-25T19:51:34.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.181+0000 7fd94d197640 10 monclient: init 2024-03-25T19:51:34.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:34.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:51:34.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-b con 0x7fd9480b1730 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:34.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-c con 0x7fd9480b4d60 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient(hunting): picked mon.noname-a con 0x7fd94814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd948143690 con 0x7fd94814be90 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9481374d0 con 0x7fd9480b1730 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9481371a0 con 0x7fd9480b4d60 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:34.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.182+0000 7fd94d197640 10 monclient(hunting): authenticate will time out at 5570.328065s 2024-03-25T19:51:34.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.195+0000 7fd94759e640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fd94814be90 0x7fd9481d7b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:44096/0 (socket says 172.21.15.130:44096) 2024-03-25T19:51:34.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.195+0000 7fd94759e640 1 -- 172.21.15.130:0/1783742461 learned_addr learned my addr 172.21.15.130:0/1783742461 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:34.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.195+0000 7fd92f7fe640 10 client.?.objecter ms_handle_connect 0x7fd94814be90 2024-03-25T19:51:34.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.195+0000 7fd92f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.197+0000 7fd92f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9480b4d60 2024-03-25T19:51:34.197 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.197+0000 7fd92f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.200+0000 7fd92f7fe640 10 client.?.objecter ms_handle_connect 0x7fd9480b1730 2024-03-25T19:51:34.199 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.200+0000 7fd92f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551224103 0 0) 0x7fd9481371a0 con 0x7fd9480b4d60 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 10 monclient(hunting): my global_id is 4658 2024-03-25T19:51:34.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.209+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd934004eb0 con 0x7fd9480b4d60 2024-03-25T19:51:34.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991452232 0 0) 0x7fd948143690 con 0x7fd94814be90 2024-03-25T19:51:34.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 10 monclient(hunting): my global_id is 4665 2024-03-25T19:51:34.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.211+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9481371a0 con 0x7fd94814be90 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510657883 0 0) 0x7fd9481374d0 con 0x7fd9480b1730 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T19:51:34.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.221+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd948143690 con 0x7fd9480b1730 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3857513419 0 0) 0x7fd934004eb0 con 0x7fd9480b4d60 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9481374d0 con 0x7fd9480b4d60 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd93c001b00 con 0x7fd9480b4d60 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.226 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.226+0000 7fd92f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2914218927 0 0) 0x7fd9481371a0 con 0x7fd94814be90 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd934004eb0 con 0x7fd94814be90 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd940001c10 con 0x7fd94814be90 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.236+0000 7fd92f7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:34.236 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:34.237 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.245+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2635396151 0 0) 0x7fd9481374d0 con 0x7fd9480b4d60 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.245+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 >> v1:172.21.15.163:6789/0 conn(0x7fd9480b1730 legacy=0x7fd948136a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 >> v1:172.21.15.130:6789/0 conn(0x7fd94814be90 legacy=0x7fd9481d7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 1 monclient: found mon.c 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd948138900 con 0x7fd9480b4d60 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:34.246 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd92f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 5 monclient: authenticate success, global_id 4658 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 10 monclient: _renew_subs 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd948135e10 con 0x7fd9480b4d60 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 10 client.4658.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 10 monclient: _renew_subs 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd948130650 con 0x7fd9480b4d60 2024-03-25T19:51:34.247 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.246+0000 7fd94d197640 1 librados: init done 2024-03-25T19:51:34.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.247+0000 7fd94d197640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.247+0000 7fd94d197640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.247+0000 7fd94d197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:34.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.247+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9481401f0 con 0x7fd9480b4d60 2024-03-25T19:51:34.561 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd93c004a90 con 0x7fd9480b4d60 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd93c004da0 con 0x7fd9480b4d60 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 10 monclient: dump: 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:34.562 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd94559a640 4 set_mon_vals no callback set 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.562+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fd93c01ad70 con 0x7fd9480b4d60 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.563+0000 7fd946d9d640 10 monclient: get_auth_request con 0x7fd93406df70 auth_method 0 2024-03-25T19:51:34.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.563+0000 7fd92f7fe640 10 client.4658.objecter ms_handle_connect 0x7fd93406df70 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fd93c0548b0 con 0x7fd9480b4d60 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 10 client.4658.objecter ms_dispatch 0x7fd948000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 3 client.4658.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 3 client.4658.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 20 client.4658.objecter dump_active .. 0 homeless 2024-03-25T19:51:34.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd93c0501f0 con 0x7fd9480b4d60 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:34.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.568+0000 7fd92f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:34.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.786+0000 7fd94d197640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:34.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.786+0000 7fd94d197640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:34.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.786+0000 7fd94d197640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:34.786 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.786+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd948137330 con 0x7fd9480b4d60 2024-03-25T19:51:34.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.850+0000 7fd92f7fe640 1 -- 172.21.15.130:0/1783742461 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fd948137330 con 0x7fd9480b4d60 2024-03-25T19:51:34.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.851+0000 7fd92f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:34.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.851+0000 7fd92f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:34.851 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:34.851 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 10 librados: watch_flush enter 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 10 librados: watch_flush exit 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 20 client.4658.objecter shutdown clearing up homeless session... 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 10 client.4658.objecter successfully canceled tick 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 >> v1:172.21.15.163:6800/1217380380 conn(0x7fd93406df70 legacy=0x7fd934070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 10 monclient: shutdown 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 >> v1:172.21.15.130:6790/0 conn(0x7fd9480b4d60 legacy=0x7fd94812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 shutdown_connections 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 >> 172.21.15.130:0/1783742461 conn(0x7fd9480617a0 msgr2=0x7fd9480b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:34.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 shutdown_connections 2024-03-25T19:51:34.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 -- 172.21.15.130:0/1783742461 wait complete. 2024-03-25T19:51:34.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:34.853+0000 7fd94d197640 1 librados: shutdown 2024-03-25T19:51:34.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:41.670 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.788+0000 7fd1f02fb640 10 monclient: get_monmap_and_config 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.788+0000 7fd1f02fb640 10 monclient: build_initial_monmap 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.788+0000 7fd1f02fb640 10 monclient: monmap: 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:41.789428+0000 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:41.789428+0000 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:41.788 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:41.789 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:41.789 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:41.789 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:41.789 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.789+0000 7fd1f02fb640 1 Processor -- start 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.789+0000 7fd1f02fb640 1 -- start start 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.789+0000 7fd1f02fb640 10 monclient: init 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e813fc60 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e8070570 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:41.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e80727a0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:41.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e813c9f0 con 0x7fd1e813fc60 2024-03-25T19:51:41.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e806fac0 con 0x7fd1e8070570 2024-03-25T19:51:41.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80715d0 con 0x7fd1e80727a0 2024-03-25T19:51:41.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:41.791 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.790+0000 7fd1f02fb640 10 monclient(hunting): authenticate will time out at 5577.935953s 2024-03-25T19:51:41.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.794+0000 7fd1ed896640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1e8070570 0x7fd1e8072090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:42818/0 (socket says 172.21.15.130:42818) 2024-03-25T19:51:41.794 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.794+0000 7fd1ed896640 1 -- 172.21.15.130:0/2303617405 learned_addr learned my addr 172.21.15.130:0/2303617405 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781380597 0 0) 0x7fd1e806fac0 con 0x7fd1e8070570 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 10 monclient(hunting): _negotiate 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 10 monclient(hunting): my global_id is 4663 2024-03-25T19:51:41.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.804+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d4003520 con 0x7fd1e8070570 2024-03-25T19:51:41.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2160879476 0 0) 0x7fd1d4003520 con 0x7fd1e8070570 2024-03-25T19:51:41.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 >> v1:172.21.15.130:6790/0 conn(0x7fd1e80727a0 legacy=0x7fd1e814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T19:51:41.810 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 >> v1:172.21.15.130:6789/0 conn(0x7fd1e813fc60 legacy=0x7fd1e806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: _finish_hunting 0 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 monclient: found mon.noname-c 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80718b0 con 0x7fd1e8070570 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: _finish_auth 0 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: _check_auth_tickets 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1d8001690 con 0x7fd1e8070570 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1ed095640 10 monclient: dump: 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.811 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1f02fb640 5 monclient: authenticate success, global_id 4663 2024-03-25T19:51:41.812 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.810+0000 7fd1f02fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1d8001860 con 0x7fd1e8070570 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 1 -- 172.21.15.130:0/2303617405 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1d8001cb0 con 0x7fd1e8070570 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1ed095640 10 monclient: dump: 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:41.830 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1f02fb640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:41.831 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.830+0000 7fd1f02fb640 4 set_mon_vals no callback set 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 10 monclient: shutdown 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2303617405 >> v1:172.21.15.163:6789/0 conn(0x7fd1e8070570 legacy=0x7fd1e8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2303617405 shutdown_connections 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2303617405 >> 172.21.15.130:0/2303617405 conn(0x7fd1e80b4a80 msgr2=0x7fd1e80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2303617405 shutdown_connections 2024-03-25T19:51:41.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.831+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2303617405 wait complete. 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 10 monclient: build_initial_monmap 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 10 monclient: monmap: 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 librados: starting msgr at 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 librados: starting objecter 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 Processor -- start 2024-03-25T19:51:41.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 -- start start 2024-03-25T19:51:41.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 librados: setting wanted keys 2024-03-25T19:51:41.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 1 librados: calling monclient init 2024-03-25T19:51:41.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.832+0000 7fd1f02fb640 10 monclient: init 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.833+0000 7fd1f02fb640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.833+0000 7fd1f02fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.833+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e8070570 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.833+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e80727a0 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e813fc60 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8138780 con 0x7fd1e80727a0 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e8135ee0 con 0x7fd1e8070570 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e81d5730 con 0x7fd1e813fc60 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:41.835 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.834+0000 7fd1f02fb640 10 monclient(hunting): authenticate will time out at 5577.979916s 2024-03-25T19:51:41.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.841+0000 7fd1ee097640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd1e8070570 0x7fd1e81d5c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:42848/0 (socket says 172.21.15.130:42848) 2024-03-25T19:51:41.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.841+0000 7fd1ee097640 1 -- 172.21.15.130:0/2164450286 learned_addr learned my addr 172.21.15.130:0/2164450286 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:41.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.842+0000 7fd1deffd640 10 client.?.objecter ms_handle_connect 0x7fd1e8070570 2024-03-25T19:51:41.841 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.842+0000 7fd1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.844+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673198788 0 0) 0x7fd1e8135ee0 con 0x7fd1e8070570 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.845+0000 7fd1deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.845+0000 7fd1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.845+0000 7fd1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.845+0000 7fd1deffd640 10 monclient(hunting): my global_id is 4666 2024-03-25T19:51:41.844 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.845+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc004eb0 con 0x7fd1e8070570 2024-03-25T19:51:41.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.846+0000 7fd1deffd640 10 client.?.objecter ms_handle_connect 0x7fd1e80727a0 2024-03-25T19:51:41.846 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.846+0000 7fd1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:41.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.850+0000 7fd1deffd640 10 client.?.objecter ms_handle_connect 0x7fd1e813fc60 2024-03-25T19:51:41.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.850+0000 7fd1deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:41.858 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4062745932 0 0) 0x7fd1cc004eb0 con 0x7fd1e8070570 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e8135ee0 con 0x7fd1e8070570 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1e4001a30 con 0x7fd1e8070570 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 10 monclient(hunting): dump: 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:41.930 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:41.931 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.859+0000 7fd1deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806367441 0 0) 0x7fd1e81d5730 con 0x7fd1e813fc60 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 10 monclient(hunting): my global_id is 4661 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.860+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc004eb0 con 0x7fd1e813fc60 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1016693677 0 0) 0x7fd1e8138780 con 0x7fd1e80727a0 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 10 monclient(hunting): _negotiate 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 10 monclient(hunting): my global_id is 4671 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.863+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e81d5730 con 0x7fd1e80727a0 2024-03-25T19:51:41.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3174879520 0 0) 0x7fd1e8135ee0 con 0x7fd1e8070570 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 >> v1:172.21.15.130:6790/0 conn(0x7fd1e813fc60 legacy=0x7fd1e812fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 >> v1:172.21.15.130:6789/0 conn(0x7fd1e80727a0 legacy=0x7fd1e8136950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 10 monclient: _finish_hunting 0 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 1 monclient: found mon.b 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e81d6360 con 0x7fd1e8070570 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 10 monclient: _finish_auth 0 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 10 monclient: _check_auth_tickets 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.865+0000 7fd1f02fb640 5 monclient: authenticate success, global_id 4666 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 10 monclient: _renew_subs 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e8133d00 con 0x7fd1e8070570 2024-03-25T19:51:41.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 10 monclient: _renew_subs 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e8137240 con 0x7fd1e8070570 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.866+0000 7fd1f02fb640 1 librados: init done 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.867+0000 7fd1f02fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.867+0000 7fd1f02fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.867+0000 7fd1f02fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:41.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:41.867+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e81385c0 con 0x7fd1e8070570 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1e4001cb0 con 0x7fd1e8070570 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd1e4003a60 con 0x7fd1e8070570 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1deffd640 10 monclient: dump: 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:42.146 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:42.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.146+0000 7fd1ec894640 4 set_mon_vals no callback set 2024-03-25T19:51:42.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.147+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fd1e4003c80 con 0x7fd1e8070570 2024-03-25T19:51:42.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.148+0000 7fd1ed896640 10 monclient: get_auth_request con 0x7fd1cc06de10 auth_method 0 2024-03-25T19:51:42.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.148+0000 7fd1deffd640 10 client.4666.objecter ms_handle_connect 0x7fd1cc06de10 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1964985173 0 0) 0x7fd1e4054990 con 0x7fd1e8070570 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 10 client.4666.objecter ms_dispatch 0x7fd1e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 3 client.4666.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 3 client.4666.objecter handle_osd_map decoding full epoch 10 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1e40502d0 con 0x7fd1e8070570 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:42.166 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.166+0000 7fd1deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:42.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.426+0000 7fd1f02fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.426+0000 7fd1f02fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.426+0000 7fd1f02fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:51:42.426 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.426+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1e81d58c0 con 0x7fd1e8070570 2024-03-25T19:51:42.475 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.476+0000 7fd1deffd640 1 -- 172.21.15.130:0/2164450286 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2331901692) 0x7fd1e40523d0 con 0x7fd1e8070570 2024-03-25T19:51:42.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.476+0000 7fd1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:42.476 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.476+0000 7fd1deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:42.476 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:42.476 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":10,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:41:48.010168+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:41:48.010168+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:42.478 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 10 librados: watch_flush enter 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 10 librados: watch_flush exit 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 10 client.4666.objecter successfully canceled tick 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 >> v1:172.21.15.163:6800/1217380380 conn(0x7fd1cc06de10 legacy=0x7fd1cc0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 10 monclient: shutdown 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 >> v1:172.21.15.163:6789/0 conn(0x7fd1e8070570 legacy=0x7fd1e81d5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 shutdown_connections 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 >> 172.21.15.130:0/2164450286 conn(0x7fd1e80b4a80 msgr2=0x7fd1e80ae910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 shutdown_connections 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 -- 172.21.15.130:0/2164450286 wait complete. 2024-03-25T19:51:42.479 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:42.478+0000 7fd1f02fb640 1 librados: shutdown 2024-03-25T19:51:42.491 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:49.297 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.412+0000 7fa10b115640 10 monclient: get_monmap_and_config 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.412+0000 7fa10b115640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.412+0000 7fa10b115640 10 monclient: monmap: 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:49.413418+0000 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:49.413418+0000 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:49.412 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:49.413 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.413+0000 7fa10b115640 1 Processor -- start 2024-03-25T19:51:49.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.413+0000 7fa10b115640 1 -- start start 2024-03-25T19:51:49.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.413+0000 7fa10b115640 10 monclient: init 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-c con 0x7fa1040b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-a con 0x7fa1040b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-b con 0x7fa10414be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040b0a70 con 0x7fa1040b4d60 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1040b0d50 con 0x7fa10414be90 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.414+0000 7fa10b115640 10 monclient(hunting): authenticate will time out at 5585.560057s 2024-03-25T19:51:49.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.424+0000 7fa108eb1640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fa1040b4d60 0x7fa10413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:40360/0 (socket says 172.21.15.130:40360) 2024-03-25T19:51:49.424 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.424+0000 7fa108eb1640 1 -- 172.21.15.130:0/492719958 learned_addr learned my addr 172.21.15.130:0/492719958 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:49.433 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562277738 0 0) 0x7fa1040b0a70 con 0x7fa1040b4d60 2024-03-25T19:51:49.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 10 monclient(hunting): my global_id is 4669 2024-03-25T19:51:49.434 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.434+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f4003520 con 0x7fa1040b4d60 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2170871423 0 0) 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 10 monclient(hunting): my global_id is 4674 2024-03-25T19:51:49.438 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.438+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1040b0a70 con 0x7fa1040b1730 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523505843 0 0) 0x7fa1040b0d50 con 0x7fa10414be90 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 10 monclient(hunting): my global_id is 4664 2024-03-25T19:51:49.459 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.459+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1040af940 con 0x7fa10414be90 2024-03-25T19:51:49.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442714252 0 0) 0x7fa0f4003520 con 0x7fa1040b4d60 2024-03-25T19:51:49.464 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 >> v1:172.21.15.130:6790/0 conn(0x7fa10414be90 legacy=0x7fa10414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 >> v1:172.21.15.130:6789/0 conn(0x7fa1040b1730 legacy=0x7fa1040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 monclient: found mon.noname-c 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1040b1030 con 0x7fa1040b4d60 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0f8001690 con 0x7fa1040b4d60 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa1037fe640 10 monclient: dump: 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.465 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa10b115640 5 monclient: authenticate success, global_id 4669 2024-03-25T19:51:49.466 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.465+0000 7fa10b115640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f80017f0 con 0x7fa1040b4d60 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 1 -- 172.21.15.130:0/492719958 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0f8001c00 con 0x7fa1040b4d60 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa1037fe640 10 monclient: dump: 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.506 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa10b115640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:49.507 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.506+0000 7fa10b115640 4 set_mon_vals no callback set 2024-03-25T19:51:49.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 10 monclient: shutdown 2024-03-25T19:51:49.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:49.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 1 -- 172.21.15.130:0/492719958 >> v1:172.21.15.163:6789/0 conn(0x7fa1040b4d60 legacy=0x7fa10413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.508 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 1 -- 172.21.15.130:0/492719958 shutdown_connections 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 1 -- 172.21.15.130:0/492719958 >> 172.21.15.130:0/492719958 conn(0x7fa1040617a0 msgr2=0x7fa104061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 1 -- 172.21.15.130:0/492719958 shutdown_connections 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.507+0000 7fa10b115640 1 -- 172.21.15.130:0/492719958 wait complete. 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.508+0000 7fa10b115640 10 monclient: build_initial_monmap 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.508+0000 7fa10b115640 10 monclient: monmap: 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:49.509 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.508+0000 7fa10b115640 1 librados: starting msgr at 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.508+0000 7fa10b115640 1 librados: starting objecter 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.509+0000 7fa10b115640 1 Processor -- start 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.509+0000 7fa10b115640 1 -- start start 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.509+0000 7fa10b115640 1 librados: setting wanted keys 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.509+0000 7fa10b115640 1 librados: calling monclient init 2024-03-25T19:51:49.510 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.509+0000 7fa10b115640 10 monclient: init 2024-03-25T19:51:49.511 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-a con 0x7fa1040b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-b con 0x7fa1040b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient(hunting): picked mon.noname-c con 0x7fa10414be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104141f40 con 0x7fa1040b1730 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa104142360 con 0x7fa1040b4d60 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1041424f0 con 0x7fa10414be90 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:49.512 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.510+0000 7fa10b115640 10 monclient(hunting): authenticate will time out at 5585.655798s 2024-03-25T19:51:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.521+0000 7fa1096b2640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fa10414be90 0x7fa1041e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51536/0 (socket says 172.21.15.130:51536) 2024-03-25T19:51:49.521 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.521+0000 7fa1096b2640 1 -- 172.21.15.130:0/1800636325 learned_addr learned my addr 172.21.15.130:0/1800636325 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:49.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.525+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa10414be90 2024-03-25T19:51:49.524 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.525+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.529+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa1040b1730 2024-03-25T19:51:49.529 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.529+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.536+0000 7fa1017fa640 10 client.?.objecter ms_handle_connect 0x7fa1040b4d60 2024-03-25T19:51:49.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.536+0000 7fa1017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:49.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778444795 0 0) 0x7fa1040af940 con 0x7fa10414be90 2024-03-25T19:51:49.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4667 2024-03-25T19:51:49.543 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.543+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8004eb0 con 0x7fa10414be90 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 232628613 0 0) 0x7fa104141f40 con 0x7fa1040b1730 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4677 2024-03-25T19:51:49.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.549+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294484327 0 0) 0x7fa104142360 con 0x7fa1040b4d60 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 10 monclient(hunting): _negotiate 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 10 monclient(hunting): my global_id is 4672 2024-03-25T19:51:49.553 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.553+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa104141f40 con 0x7fa1040b4d60 2024-03-25T19:51:49.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3241778796 0 0) 0x7fa0e8004eb0 con 0x7fa10414be90 2024-03-25T19:51:49.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa104142360 con 0x7fa10414be90 2024-03-25T19:51:49.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0fc001ad0 con 0x7fa10414be90 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.575+0000 7fa1017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:49.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2845270402 0 0) 0x7fa1040af940 con 0x7fa1040b1730 2024-03-25T19:51:49.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e8004eb0 con 0x7fa1040b1730 2024-03-25T19:51:49.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0f8001b00 con 0x7fa1040b1730 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.577+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.577 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3241944769 0 0) 0x7fa104141f40 con 0x7fa1040b4d60 2024-03-25T19:51:49.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1040af940 con 0x7fa1040b4d60 2024-03-25T19:51:49.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0f0001930 con 0x7fa1040b4d60 2024-03-25T19:51:49.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.583+0000 7fa1017fa640 10 monclient(hunting): dump: 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.583 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2703680292 0 0) 0x7fa104142360 con 0x7fa10414be90 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 >> v1:172.21.15.163:6789/0 conn(0x7fa1040b4d60 legacy=0x7fa1041d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 >> v1:172.21.15.130:6789/0 conn(0x7fa1040b1730 legacy=0x7fa1041405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 10 monclient: _finish_hunting 0 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 1 monclient: found mon.c 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104140d00 con 0x7fa10414be90 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 10 monclient: _finish_auth 0 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 10 monclient: _check_auth_tickets 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.595+0000 7fa1017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 5 monclient: authenticate success, global_id 4667 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 10 monclient: _renew_subs 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:49.596 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa104138150 con 0x7fa10414be90 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 10 monclient: _renew_subs 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1041d8520 con 0x7fa10414be90 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.596+0000 7fa10b115640 1 librados: init done 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.597+0000 7fa10b115640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:49.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.597+0000 7fa10b115640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:49.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.597+0000 7fa10b115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:49.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.597+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa104140f50 con 0x7fa10414be90 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0fc001fe0 con 0x7fa10414be90 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fa0fc003b30 con 0x7fa10414be90 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:49.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa1017fa640 10 monclient: dump: 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:49.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.938+0000 7fa102ffd640 4 set_mon_vals no callback set 2024-03-25T19:51:49.940 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.939+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fa0fc003d50 con 0x7fa10414be90 2024-03-25T19:51:49.940 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.940+0000 7fa108eb1640 10 monclient: get_auth_request con 0x7fa0e806e0b0 auth_method 0 2024-03-25T19:51:49.940 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.940+0000 7fa1017fa640 10 client.4667.objecter ms_handle_connect 0x7fa0e806e0b0 2024-03-25T19:51:49.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3677576774 0 0) 0x7fa0fc0548b0 con 0x7fa10414be90 2024-03-25T19:51:49.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 10 client.4667.objecter ms_dispatch 0x7fa104000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0fc0503c0 con 0x7fa10414be90 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:49.946 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:49.946+0000 7fa1017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:50.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.183+0000 7fa10b115640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.183+0000 7fa10b115640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.183+0000 7fa10b115640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:50.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.183+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa104138450 con 0x7fa10414be90 2024-03-25T19:51:50.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.210+0000 7fa1017fa640 1 -- 172.21.15.130:0/1800636325 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7086 (unknown 238102980 0 2986191470) 0x7fa0fc0522f0 con 0x7fa10414be90 2024-03-25T19:51:50.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.210+0000 7fa1017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:50.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.210+0000 7fa1017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:50.210 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:50.210 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":11,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:45.684257+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:45.684257+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:50.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.212+0000 7fa10b115640 10 librados: watch_flush enter 2024-03-25T19:51:50.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.212+0000 7fa10b115640 10 librados: watch_flush exit 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.212+0000 7fa10b115640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.212+0000 7fa10b115640 10 client.4667.objecter successfully canceled tick 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 >> v1:172.21.15.163:6800/1217380380 conn(0x7fa0e806e0b0 legacy=0x7fa0e8070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 10 monclient: shutdown 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 >> v1:172.21.15.130:6790/0 conn(0x7fa10414be90 legacy=0x7fa1041e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 shutdown_connections 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 >> 172.21.15.130:0/1800636325 conn(0x7fa1040617a0 msgr2=0x7fa1040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 shutdown_connections 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 -- 172.21.15.130:0/1800636325 wait complete. 2024-03-25T19:51:50.213 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:50.213+0000 7fa10b115640 1 librados: shutdown 2024-03-25T19:51:50.223 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:51:50.976 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:35.908 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:50.981 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:35.914 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.029 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:35.962 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.032 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:35.965 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.053 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:35.986 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.057 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:35.990 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.096 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:35.997 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.106 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:36.007 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:51.127 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:36.028 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:51:57.028 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.144+0000 7fecc936b640 10 monclient: get_monmap_and_config 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.144+0000 7fecc936b640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.144+0000 7fecc936b640 10 monclient: monmap: 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:51:57.146337+0000 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:51:57.146337+0000 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:57.145 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:51:57.146 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:51:57.146 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.145+0000 7fecc936b640 1 Processor -- start 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.145+0000 7fecc936b640 1 -- start start 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.145+0000 7fecc936b640 10 monclient: init 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-b con 0x7fecc40a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-c con 0x7fecc413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-a con 0x7fecc41430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:57.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc4142520 con 0x7fecc41430c0 2024-03-25T19:51:57.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc40b28a0 con 0x7fecc413f710 2024-03-25T19:51:57.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc40b2a30 con 0x7fecc40a98f0 2024-03-25T19:51:57.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.146+0000 7fecc936b640 10 monclient(hunting): authenticate will time out at 5593.293155s 2024-03-25T19:51:57.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.156+0000 7fecc2ffd640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fecc40a98f0 0x7fecc40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:58604/0 (socket says 172.21.15.130:58604) 2024-03-25T19:51:57.157 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.156+0000 7fecc2ffd640 1 -- 172.21.15.130:0/884929987 learned_addr learned my addr 172.21.15.130:0/884929987 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675659386 0 0) 0x7fecc40b2a30 con 0x7fecc40a98f0 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): my global_id is 4670 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecac003520 con 0x7fecc40a98f0 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578495874 0 0) 0x7fecc4142520 con 0x7fecc41430c0 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 10 monclient(hunting): my global_id is 4680 2024-03-25T19:51:57.171 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.170+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc40b2a30 con 0x7fecc41430c0 2024-03-25T19:51:57.181 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3196519711 0 0) 0x7fecac003520 con 0x7fecc40a98f0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 >> v1:172.21.15.163:6789/0 conn(0x7fecc413f710 legacy=0x7fecc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 >> v1:172.21.15.130:6789/0 conn(0x7fecc41430c0 legacy=0x7fecc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 monclient: found mon.noname-b 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc40b44c0 con 0x7fecc40a98f0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fecb4001690 con 0x7fecc40a98f0 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc1ffb640 10 monclient: dump: 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc936b640 5 monclient: authenticate success, global_id 4670 2024-03-25T19:51:57.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.181+0000 7fecc936b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecb40017f0 con 0x7fecc40a98f0 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 1 -- 172.21.15.130:0/884929987 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fecb4001c00 con 0x7fecc40a98f0 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc936b640 10 monclient: get_monmap_and_config success 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc936b640 4 set_mon_vals no callback set 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.188+0000 7fecc1ffb640 10 monclient: dump: 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.189 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.190 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:57.190 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:57.190 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:57.190 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:57.190 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 10 monclient: shutdown 2024-03-25T19:51:57.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:57.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 1 -- 172.21.15.130:0/884929987 >> v1:172.21.15.130:6790/0 conn(0x7fecc40a98f0 legacy=0x7fecc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.191 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 1 -- 172.21.15.130:0/884929987 shutdown_connections 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 1 -- 172.21.15.130:0/884929987 >> 172.21.15.130:0/884929987 conn(0x7fecc40b7ea0 msgr2=0x7fecc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 1 -- 172.21.15.130:0/884929987 shutdown_connections 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.189+0000 7fecc936b640 1 -- 172.21.15.130:0/884929987 wait complete. 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 10 monclient: build_initial_monmap 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 10 monclient: monmap: 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:51:57.192 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 librados: starting msgr at 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 librados: starting objecter 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 Processor -- start 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 -- start start 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 librados: setting wanted keys 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 1 librados: calling monclient init 2024-03-25T19:51:57.193 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.190+0000 7fecc936b640 10 monclient: init 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient: _reopen_session rank -1 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-a con 0x7fecc40a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-c con 0x7fecc413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient(hunting): picked mon.noname-b con 0x7fecc41430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc41370a0 con 0x7fecc40a98f0 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc4135090 con 0x7fecc41430c0 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fecc4135220 con 0x7fecc413f710 2024-03-25T19:51:57.194 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient(hunting): _renew_subs 2024-03-25T19:51:57.195 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.191+0000 7fecc936b640 10 monclient(hunting): authenticate will time out at 5593.338359s 2024-03-25T19:51:57.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.199+0000 7fecc27fc640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fecc413f710 0x7fecc41d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:58640/0 (socket says 172.21.15.130:58640) 2024-03-25T19:51:57.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.199+0000 7fecc27fc640 1 -- 172.21.15.130:0/2187073588 learned_addr learned my addr 172.21.15.130:0/2187073588 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:51:57.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.200+0000 7fecab7fe640 10 client.?.objecter ms_handle_connect 0x7fecc413f710 2024-03-25T19:51:57.200 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.200+0000 7fecab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1232902120 0 0) 0x7fecc4135220 con 0x7fecc413f710 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 monclient(hunting): my global_id is 4673 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0004eb0 con 0x7fecc413f710 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 client.?.objecter ms_handle_connect 0x7fecc41430c0 2024-03-25T19:51:57.206 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.205+0000 7fecab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.212+0000 7fecab7fe640 10 client.?.objecter ms_handle_connect 0x7fecc40a98f0 2024-03-25T19:51:57.212 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.212+0000 7fecab7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700236174 0 0) 0x7fecb0004eb0 con 0x7fecc413f710 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fecc4135220 con 0x7fecc413f710 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fecb80019d0 con 0x7fecc413f710 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 10 monclient(hunting): dump: 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.216 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:57.217 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:57.217 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:57.217 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:57.217 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.217 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.215+0000 7fecab7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771323537 0 0) 0x7fecc4135090 con 0x7fecc41430c0 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 10 monclient(hunting): my global_id is 4678 2024-03-25T19:51:57.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.219+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecb0004eb0 con 0x7fecc41430c0 2024-03-25T19:51:57.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3392059775 0 0) 0x7fecc41370a0 con 0x7fecc40a98f0 2024-03-25T19:51:57.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:51:57.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:51:57.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:51:57.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 10 monclient(hunting): my global_id is 4683 2024-03-25T19:51:57.222 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.221+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fecc4135090 con 0x7fecc40a98f0 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3875079429 0 0) 0x7fecc4135220 con 0x7fecc413f710 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 >> v1:172.21.15.163:6789/0 conn(0x7fecc41430c0 legacy=0x7fecc41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 >> v1:172.21.15.130:6789/0 conn(0x7fecc40a98f0 legacy=0x7fecc41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 1 monclient: found mon.c 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc41d5000 con 0x7fecc413f710 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 10 monclient: _finish_auth 0 2024-03-25T19:51:57.227 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 5 monclient: authenticate success, global_id 4673 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 10 monclient: _renew_subs 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc41d52b0 con 0x7fecc413f710 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 10 monclient: _renew_subs 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc4135910 con 0x7fecc413f710 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.226+0000 7fecc936b640 1 librados: init done 2024-03-25T19:51:57.228 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.227+0000 7fecc936b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.227+0000 7fecc936b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.227+0000 7fecc936b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.229 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.227+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecc40627b0 con 0x7fecc413f710 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fecb8001c50 con 0x7fecc413f710 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fecb8003a60 con 0x7fecc413f710 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecab7fe640 10 monclient: dump: 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:51:57.539 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:51:57.540 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.538+0000 7fecc17fa640 4 set_mon_vals no callback set 2024-03-25T19:51:57.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.539+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fecb8003c80 con 0x7fecc413f710 2024-03-25T19:51:57.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.540+0000 7fecc2ffd640 10 monclient: get_auth_request con 0x7fecb006de60 auth_method 0 2024-03-25T19:51:57.541 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.540+0000 7fecab7fe640 10 client.4673.objecter ms_handle_connect 0x7fecb006de60 2024-03-25T19:51:57.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7fecb80547d0 con 0x7fecc413f710 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 10 client.4673.objecter ms_dispatch 0x7fecc4000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 3 client.4673.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 3 client.4673.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fecb80502e0 con 0x7fecc413f710 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:51:57.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.547+0000 7fecab7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:51:57.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.795+0000 7fecc936b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:57.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.795+0000 7fecc936b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:57.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.796+0000 7fecc936b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:51:57.796 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.796+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fecc40008d0 con 0x7fecc413f710 2024-03-25T19:51:57.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.830+0000 7fecab7fe640 1 -- 172.21.15.130:0/2187073588 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7fecb8052210 con 0x7fecc413f710 2024-03-25T19:51:57.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.830+0000 7fecab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:51:57.830 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.830+0000 7fecab7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:51:57.831 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:51:57.831 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 10 librados: watch_flush enter 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 10 librados: watch_flush exit 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 10 client.4673.objecter successfully canceled tick 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 >> v1:172.21.15.163:6800/1217380380 conn(0x7fecb006de60 legacy=0x7fecb0070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 10 monclient: shutdown 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 >> v1:172.21.15.130:6790/0 conn(0x7fecc413f710 legacy=0x7fecc41d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 shutdown_connections 2024-03-25T19:51:57.833 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 >> 172.21.15.130:0/2187073588 conn(0x7fecc40b7ea0 msgr2=0x7fecc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:51:57.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 shutdown_connections 2024-03-25T19:51:57.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 -- 172.21.15.130:0/2187073588 wait complete. 2024-03-25T19:51:57.834 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:51:57.832+0000 7fecc936b640 1 librados: shutdown 2024-03-25T19:51:57.845 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:04.652 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.769+0000 7fed2ba6a640 10 monclient: get_monmap_and_config 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.769+0000 7fed2ba6a640 10 monclient: build_initial_monmap 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.769+0000 7fed2ba6a640 10 monclient: monmap: 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:04.771415+0000 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:04.771415+0000 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:04.770 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.770+0000 7fed2ba6a640 1 Processor -- start 2024-03-25T19:52:04.771 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.770+0000 7fed2ba6a640 1 -- start start 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.770+0000 7fed2ba6a640 10 monclient: init 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-b con 0x7fed240a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-c con 0x7fed2413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-a con 0x7fed241430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed24142520 con 0x7fed241430c0 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240b28a0 con 0x7fed2413f710 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed240b2a30 con 0x7fed240a98f0 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:04.772 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.771+0000 7fed2ba6a640 10 monclient(hunting): authenticate will time out at 5600.918222s 2024-03-25T19:52:04.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.790+0000 7fed29005640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fed2413f710 0x7fed241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:54060/0 (socket says 172.21.15.130:54060) 2024-03-25T19:52:04.790 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.790+0000 7fed29005640 1 -- 172.21.15.130:0/94583176 learned_addr learned my addr 172.21.15.130:0/94583176 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:04.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.803+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402593944 0 0) 0x7fed240b2a30 con 0x7fed240a98f0 2024-03-25T19:52:04.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.804+0000 7fed28804640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.804+0000 7fed28804640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.804+0000 7fed28804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.804 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.804+0000 7fed28804640 10 monclient(hunting): my global_id is 4676 2024-03-25T19:52:04.805 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.804+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed10003520 con 0x7fed240a98f0 2024-03-25T19:52:04.808 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687085750 0 0) 0x7fed24142520 con 0x7fed241430c0 2024-03-25T19:52:04.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 10 monclient(hunting): my global_id is 4686 2024-03-25T19:52:04.809 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.808+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed240b2a30 con 0x7fed241430c0 2024-03-25T19:52:04.818 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223121167 0 0) 0x7fed240b28a0 con 0x7fed2413f710 2024-03-25T19:52:04.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 10 monclient(hunting): my global_id is 4681 2024-03-25T19:52:04.819 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.818+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24142520 con 0x7fed2413f710 2024-03-25T19:52:04.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 801661582 0 0) 0x7fed10003520 con 0x7fed240a98f0 2024-03-25T19:52:04.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 >> v1:172.21.15.163:6789/0 conn(0x7fed2413f710 legacy=0x7fed241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:04.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 >> v1:172.21.15.130:6789/0 conn(0x7fed241430c0 legacy=0x7fed2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:04.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: _finish_hunting 0 2024-03-25T19:52:04.827 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 monclient: found mon.noname-b 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed240b44c0 con 0x7fed240a98f0 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: _finish_auth 0 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: _check_auth_tickets 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fed20001690 con 0x7fed240a98f0 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed2ba6a640 5 monclient: authenticate success, global_id 4676 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed2ba6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed28804640 10 monclient: dump: 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:04.828 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:04.829 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.826+0000 7fed2ba6a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed200017f0 con 0x7fed240a98f0 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 1 -- 172.21.15.130:0/94583176 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fed20001c00 con 0x7fed240a98f0 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed2ba6a640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed2ba6a640 4 set_mon_vals no callback set 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.864+0000 7fed28804640 10 monclient: dump: 2024-03-25T19:52:04.865 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:04.866 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:04.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 10 monclient: shutdown 2024-03-25T19:52:04.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:04.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 1 -- 172.21.15.130:0/94583176 >> v1:172.21.15.130:6790/0 conn(0x7fed240a98f0 legacy=0x7fed240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:04.868 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 1 -- 172.21.15.130:0/94583176 shutdown_connections 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 1 -- 172.21.15.130:0/94583176 >> 172.21.15.130:0/94583176 conn(0x7fed240b7ea0 msgr2=0x7fed240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 1 -- 172.21.15.130:0/94583176 shutdown_connections 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.865+0000 7fed2ba6a640 1 -- 172.21.15.130:0/94583176 wait complete. 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.866+0000 7fed2ba6a640 10 monclient: build_initial_monmap 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.866+0000 7fed2ba6a640 10 monclient: monmap: 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:04.869 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.866+0000 7fed2ba6a640 1 librados: starting msgr at 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.866+0000 7fed2ba6a640 1 librados: starting objecter 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 Processor -- start 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 -- start start 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 librados: setting wanted keys 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 librados: calling monclient init 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient: init 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-b con 0x7fed240a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-c con 0x7fed2413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient(hunting): picked mon.noname-a con 0x7fed241430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2406e9d0 con 0x7fed241430c0 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2406ec40 con 0x7fed240a98f0 2024-03-25T19:52:04.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed2406ef20 con 0x7fed2413f710 2024-03-25T19:52:04.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:04.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.867+0000 7fed2ba6a640 10 monclient(hunting): authenticate will time out at 5601.014439s 2024-03-25T19:52:04.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.878+0000 7fed29806640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fed240a98f0 0x7fed2413b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:54148/0 (socket says 172.21.15.130:54148) 2024-03-25T19:52:04.878 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.878+0000 7fed29806640 1 -- 172.21.15.130:0/3430036105 learned_addr learned my addr 172.21.15.130:0/3430036105 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:04.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.878+0000 7fed1a7fc640 10 client.?.objecter ms_handle_connect 0x7fed240a98f0 2024-03-25T19:52:04.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.878+0000 7fed1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:04.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 client.?.objecter ms_handle_connect 0x7fed241430c0 2024-03-25T19:52:04.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:04.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1803091619 0 0) 0x7fed2406ec40 con 0x7fed240a98f0 2024-03-25T19:52:04.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.886 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 10 monclient(hunting): my global_id is 4684 2024-03-25T19:52:04.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.885+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed08004eb0 con 0x7fed240a98f0 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111344109 0 0) 0x7fed2406e9d0 con 0x7fed241430c0 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 10 monclient(hunting): my global_id is 4689 2024-03-25T19:52:04.893 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.892+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2406ec40 con 0x7fed241430c0 2024-03-25T19:52:04.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.894+0000 7fed1a7fc640 10 client.?.objecter ms_handle_connect 0x7fed2413f710 2024-03-25T19:52:04.894 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.894+0000 7fed1a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:04.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4033784739 0 0) 0x7fed2406ef20 con 0x7fed2413f710 2024-03-25T19:52:04.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 10 monclient(hunting): _negotiate 2024-03-25T19:52:04.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:04.906 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:04.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 10 monclient(hunting): my global_id is 4679 2024-03-25T19:52:04.907 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.906+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed2406e9d0 con 0x7fed2413f710 2024-03-25T19:52:04.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17468765 0 0) 0x7fed08004eb0 con 0x7fed240a98f0 2024-03-25T19:52:04.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed2406ef20 con 0x7fed240a98f0 2024-03-25T19:52:04.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fed20001bc0 con 0x7fed240a98f0 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 10 monclient(hunting): dump: 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:04.910 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:04.911 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:04.911 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:04.911 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.909+0000 7fed1a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:04.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 317008268 0 0) 0x7fed2406ef20 con 0x7fed240a98f0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 >> v1:172.21.15.130:6790/0 conn(0x7fed2413f710 legacy=0x7fed2406f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 >> v1:172.21.15.130:6789/0 conn(0x7fed241430c0 legacy=0x7fed241e64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 10 monclient: _finish_hunting 0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 1 monclient: found mon.b 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2406fda0 con 0x7fed240a98f0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 10 monclient: _finish_auth 0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 10 monclient: _check_auth_tickets 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.915+0000 7fed2ba6a640 5 monclient: authenticate success, global_id 4684 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _renew_subs 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed2414dad0 con 0x7fed240a98f0 2024-03-25T19:52:04.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _renew_subs 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fed2406e110 con 0x7fed240a98f0 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 1 librados: init done 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:04.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:04.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:04.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:04.916+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed240627b0 con 0x7fed240a98f0 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed200011f0 con 0x7fed240a98f0 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fed20003960 con 0x7fed240a98f0 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1bfff640 4 set_mon_vals no callback set 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.230+0000 7fed1a7fc640 10 monclient: dump: 2024-03-25T19:52:05.231 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:05.232 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:05.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.231+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fed20003b80 con 0x7fed240a98f0 2024-03-25T19:52:05.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.232+0000 7fed29005640 10 monclient: get_auth_request con 0x7fed0806dd50 auth_method 0 2024-03-25T19:52:05.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.232+0000 7fed1a7fc640 10 client.4684.objecter ms_handle_connect 0x7fed0806dd50 2024-03-25T19:52:05.236 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7fed200546f0 con 0x7fed240a98f0 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 10 client.4684.objecter ms_dispatch 0x7fed24000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 3 client.4684.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 3 client.4684.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fed20050200 con 0x7fed240a98f0 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:05.237 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.236+0000 7fed1a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:05.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.479+0000 7fed2ba6a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.480+0000 7fed2ba6a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.480+0000 7fed2ba6a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:05.480 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.480+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed240b3e80 con 0x7fed240a98f0 2024-03-25T19:52:05.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.609+0000 7fed1a7fc640 1 -- 172.21.15.130:0/3430036105 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7fed20052130 con 0x7fed240a98f0 2024-03-25T19:52:05.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.609+0000 7fed1a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:05.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.609+0000 7fed1a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:05.610 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:05.610 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:05.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 10 librados: watch_flush enter 2024-03-25T19:52:05.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 10 librados: watch_flush exit 2024-03-25T19:52:05.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T19:52:05.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 10 client.4684.objecter successfully canceled tick 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 >> v1:172.21.15.163:6800/1217380380 conn(0x7fed0806dd50 legacy=0x7fed08070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 10 monclient: shutdown 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 >> v1:172.21.15.163:6789/0 conn(0x7fed240a98f0 legacy=0x7fed2413b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 shutdown_connections 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 >> 172.21.15.130:0/3430036105 conn(0x7fed240b7ea0 msgr2=0x7fed24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 shutdown_connections 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 -- 172.21.15.130:0/3430036105 wait complete. 2024-03-25T19:52:05.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:05.611+0000 7fed2ba6a640 1 librados: shutdown 2024-03-25T19:52:05.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:10.932 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:55.864 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:10.940 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:55.873 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:10.958 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:10:55.891 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:10.988 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:55.889 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:10.995 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:55.928 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:11.000 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:55.901 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:11.014 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:55.948 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:11.035 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:10:55.936 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:11.040 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:10:55.973 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:12.431 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:12.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.547+0000 7f9f3a5ed640 10 monclient: get_monmap_and_config 2024-03-25T19:52:12.547 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.547+0000 7f9f3a5ed640 10 monclient: build_initial_monmap 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.547+0000 7f9f3a5ed640 10 monclient: monmap: 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:12.548592+0000 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:12.548592+0000 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:12.548 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.548+0000 7f9f3a5ed640 1 Processor -- start 2024-03-25T19:52:12.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.548+0000 7f9f3a5ed640 1 -- start start 2024-03-25T19:52:12.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.548+0000 7f9f3a5ed640 10 monclient: init 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9f340a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9f3413f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9f341430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f34142520 con 0x7f9f3413f710 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340b28a0 con 0x7f9f341430c0 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340b2a30 con 0x7f9f340a98f0 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:12.550 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.549+0000 7f9f3a5ed640 10 monclient(hunting): authenticate will time out at 5608.695320s 2024-03-25T19:52:12.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.555+0000 7f9f33fff640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f9f340a98f0 0x7f9f340a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:38740/0 (socket says 172.21.15.130:38740) 2024-03-25T19:52:12.556 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.555+0000 7f9f33fff640 1 -- 172.21.15.130:0/900849672 learned_addr learned my addr 172.21.15.130:0/900849672 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946910031 0 0) 0x7f9f340b2a30 con 0x7f9f340a98f0 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 10 monclient(hunting): my global_id is 4682 2024-03-25T19:52:12.566 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.566+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c003520 con 0x7f9f340a98f0 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452077034 0 0) 0x7f9f340b28a0 con 0x7f9f341430c0 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 10 monclient(hunting): my global_id is 4687 2024-03-25T19:52:12.571 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.571+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f340b2a30 con 0x7f9f341430c0 2024-03-25T19:52:12.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1835950767 0 0) 0x7f9f1c003520 con 0x7f9f340a98f0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 >> v1:172.21.15.163:6789/0 conn(0x7f9f341430c0 legacy=0x7f9f3414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 >> v1:172.21.15.130:6789/0 conn(0x7f9f3413f710 legacy=0x7f9f341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 monclient: found mon.noname-b 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340b44c0 con 0x7f9f340a98f0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f24001690 con 0x7f9f340a98f0 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f32ffd640 10 monclient: dump: 2024-03-25T19:52:12.575 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f3a5ed640 5 monclient: authenticate success, global_id 4682 2024-03-25T19:52:12.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.574+0000 7f9f3a5ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f24001870 con 0x7f9f340a98f0 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 1 -- 172.21.15.130:0/900849672 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f24001c80 con 0x7f9f340a98f0 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f3a5ed640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f3a5ed640 4 set_mon_vals no callback set 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.585+0000 7f9f32ffd640 10 monclient: dump: 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.585 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.586 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:12.586 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:12.586 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:12.586 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:12.586 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 10 monclient: shutdown 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/900849672 >> v1:172.21.15.130:6790/0 conn(0x7f9f340a98f0 legacy=0x7f9f340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/900849672 shutdown_connections 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/900849672 >> 172.21.15.130:0/900849672 conn(0x7f9f340b7ea0 msgr2=0x7f9f340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/900849672 shutdown_connections 2024-03-25T19:52:12.587 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.586+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/900849672 wait complete. 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.587+0000 7f9f3a5ed640 10 monclient: build_initial_monmap 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.587+0000 7f9f3a5ed640 10 monclient: monmap: 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.587+0000 7f9f3a5ed640 1 librados: starting msgr at 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.587+0000 7f9f3a5ed640 1 librados: starting objecter 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.587+0000 7f9f3a5ed640 1 Processor -- start 2024-03-25T19:52:12.588 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 -- start start 2024-03-25T19:52:12.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 librados: setting wanted keys 2024-03-25T19:52:12.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 librados: calling monclient init 2024-03-25T19:52:12.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient: init 2024-03-25T19:52:12.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:12.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9f340a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9f3413f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9f341430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f340b72c0 con 0x7f9f341430c0 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3413b110 con 0x7f9f340a98f0 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3413b380 con 0x7f9f3413f710 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:12.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.588+0000 7f9f3a5ed640 10 monclient(hunting): authenticate will time out at 5608.734351s 2024-03-25T19:52:12.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.595+0000 7f9f38b8a640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f9f341430c0 0x7f9f341e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:49418/0 (socket says 172.21.15.130:49418) 2024-03-25T19:52:12.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.595+0000 7f9f38b8a640 1 -- 172.21.15.130:0/2372533022 learned_addr learned my addr 172.21.15.130:0/2372533022 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:12.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.599+0000 7f9f30ff9640 10 client.?.objecter ms_handle_connect 0x7f9f341430c0 2024-03-25T19:52:12.599 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.599+0000 7f9f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.604+0000 7f9f30ff9640 10 client.?.objecter ms_handle_connect 0x7f9f340a98f0 2024-03-25T19:52:12.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.604+0000 7f9f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.610 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.610+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 72049048 0 0) 0x7f9f340b2a30 con 0x7f9f341430c0 2024-03-25T19:52:12.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.610+0000 7f9f30ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.611+0000 7f9f30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.611+0000 7f9f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.611+0000 7f9f30ff9640 10 monclient(hunting): my global_id is 4695 2024-03-25T19:52:12.611 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.611+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f341430c0 2024-03-25T19:52:12.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.613+0000 7f9f30ff9640 10 client.?.objecter ms_handle_connect 0x7f9f3413f710 2024-03-25T19:52:12.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.613+0000 7f9f30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:12.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737061402 0 0) 0x7f9f3413b110 con 0x7f9f340a98f0 2024-03-25T19:52:12.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 10 monclient(hunting): my global_id is 4690 2024-03-25T19:52:12.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.616+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f340b2a30 con 0x7f9f340a98f0 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1164215128 0 0) 0x7f9f20004eb0 con 0x7f9f341430c0 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9f3413b110 con 0x7f9f341430c0 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f2c001af0 con 0x7f9f341430c0 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:12.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.618+0000 7f9f30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008877394 0 0) 0x7f9f3413b380 con 0x7f9f3413f710 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 10 monclient(hunting): my global_id is 4685 2024-03-25T19:52:12.621 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.621+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f20004eb0 con 0x7f9f3413f710 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1585711727 0 0) 0x7f9f3413b110 con 0x7f9f341430c0 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 >> v1:172.21.15.130:6790/0 conn(0x7f9f3413f710 legacy=0x7f9f3412f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 >> v1:172.21.15.163:6789/0 conn(0x7f9f340a98f0 legacy=0x7f9f3413b7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 1 monclient: found mon.a 2024-03-25T19:52:12.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f340b7970 con 0x7f9f341430c0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 10 monclient: _finish_auth 0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 5 monclient: authenticate success, global_id 4695 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 10 monclient: _renew_subs 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f34139810 con 0x7f9f341430c0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 10 monclient: _renew_subs 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f34139f00 con 0x7f9f341430c0 2024-03-25T19:52:12.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.625+0000 7f9f3a5ed640 1 librados: init done 2024-03-25T19:52:12.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.626+0000 7f9f3a5ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:12.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.626+0000 7f9f3a5ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:12.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.626+0000 7f9f3a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:12.627 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.626+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f340b7bc0 con 0x7f9f341430c0 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9f2c001fe0 con 0x7f9f341430c0 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9f2c003b30 con 0x7f9f341430c0 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f30ff9640 10 monclient: dump: 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:12.917 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:12.918 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.917+0000 7f9f327fc640 4 set_mon_vals no callback set 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.918+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f9f2c003d50 con 0x7f9f341430c0 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.918+0000 7f9f33fff640 10 monclient: get_auth_request con 0x7f9f2006de00 auth_method 0 2024-03-25T19:52:12.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.919+0000 7f9f30ff9640 10 client.4695.objecter ms_handle_connect 0x7f9f2006de00 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f9f2c054950 con 0x7f9f341430c0 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 10 client.4695.objecter ms_dispatch 0x7f9f34000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 3 client.4695.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 3 client.4695.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9f2c050460 con 0x7f9f341430c0 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:12.924 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:12.924+0000 7f9f30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.186+0000 7f9f3a5ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.186+0000 7f9f3a5ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.186 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.186+0000 7f9f3a5ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:13.187 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.186+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9f340627b0 con 0x7f9f341430c0 2024-03-25T19:52:13.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.206+0000 7f9f30ff9640 1 -- 172.21.15.130:0/2372533022 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f9f2c052390 con 0x7f9f341430c0 2024-03-25T19:52:13.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.206+0000 7f9f30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:13.205 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.206+0000 7f9f30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:13.206 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:13.206 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 10 librados: watch_flush enter 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 10 librados: watch_flush exit 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 10 client.4695.objecter successfully canceled tick 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 >> v1:172.21.15.163:6800/1217380380 conn(0x7f9f2006de00 legacy=0x7f9f200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 10 monclient: shutdown 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 >> v1:172.21.15.130:6789/0 conn(0x7f9f341430c0 legacy=0x7f9f341e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 shutdown_connections 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.208+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 >> 172.21.15.130:0/2372533022 conn(0x7f9f340b7ea0 msgr2=0x7f9f3413e700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.209+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 shutdown_connections 2024-03-25T19:52:13.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.209+0000 7f9f3a5ed640 1 -- 172.21.15.130:0/2372533022 wait complete. 2024-03-25T19:52:13.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:13.209+0000 7f9f3a5ed640 1 librados: shutdown 2024-03-25T19:52:13.221 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:20.027 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:20.144 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.145+0000 7f8790f3a640 10 monclient: get_monmap_and_config 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.145+0000 7f8790f3a640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.145+0000 7f8790f3a640 10 monclient: monmap: 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:20.146019+0000 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:20.146019+0000 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:20.145 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:20.146 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.145+0000 7f8790f3a640 1 Processor -- start 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 1 -- start start 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient: init 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f878c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f878c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f878c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:20.147 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c142520 con 0x7f878c13f710 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0b28a0 con 0x7f878c0a98f0 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c0b2a30 con 0x7f878c1430c0 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.146+0000 7f8790f3a640 10 monclient(hunting): authenticate will time out at 5616.292433s 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.147+0000 7f878a59c640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7f878c0a98f0 0x7f878c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:51006/0 (socket says 172.21.15.130:51006) 2024-03-25T19:52:20.148 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.147+0000 7f878a59c640 1 -- 172.21.15.130:0/3906206549 learned_addr learned my addr 172.21.15.130:0/3906206549 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672829531 0 0) 0x7f878c0b2a30 con 0x7f878c1430c0 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 10 monclient(hunting): my global_id is 4688 2024-03-25T19:52:20.203 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.203+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8778003520 con 0x7f878c1430c0 2024-03-25T19:52:20.208 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1324141598 0 0) 0x7f8778003520 con 0x7f878c1430c0 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 >> v1:172.21.15.163:6789/0 conn(0x7f878c0a98f0 legacy=0x7f878c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 >> v1:172.21.15.130:6789/0 conn(0x7f878c13f710 legacy=0x7f878c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 monclient: found mon.noname-b 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c0b44c0 con 0x7f878c1430c0 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.209 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8784001a10 con 0x7f878c1430c0 2024-03-25T19:52:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8790f3a640 5 monclient: authenticate success, global_id 4688 2024-03-25T19:52:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8790f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:20.210 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8789d9b640 10 monclient: dump: 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.211 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.209+0000 7f8790f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8784001b70 con 0x7f878c1430c0 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 1 -- 172.21.15.130:0/3906206549 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8784001f80 con 0x7f878c1430c0 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8790f3a640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8790f3a640 4 set_mon_vals no callback set 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.215+0000 7f8789d9b640 10 monclient: dump: 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:20.215 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:20.216 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 10 monclient: shutdown 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 1 -- 172.21.15.130:0/3906206549 >> v1:172.21.15.130:6790/0 conn(0x7f878c1430c0 legacy=0x7f878c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 1 -- 172.21.15.130:0/3906206549 shutdown_connections 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 1 -- 172.21.15.130:0/3906206549 >> 172.21.15.130:0/3906206549 conn(0x7f878c0b7ea0 msgr2=0x7f878c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 1 -- 172.21.15.130:0/3906206549 shutdown_connections 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.216+0000 7f8790f3a640 1 -- 172.21.15.130:0/3906206549 wait complete. 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.217+0000 7f8790f3a640 10 monclient: build_initial_monmap 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.217+0000 7f8790f3a640 10 monclient: monmap: 2024-03-25T19:52:20.218 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.217+0000 7f8790f3a640 1 librados: starting msgr at 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.217+0000 7f8790f3a640 1 librados: starting objecter 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.217+0000 7f8790f3a640 1 Processor -- start 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 -- start start 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 librados: setting wanted keys 2024-03-25T19:52:20.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 librados: calling monclient init 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient: init 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f878c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f878c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f878c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c137c60 con 0x7f878c0a98f0 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c139160 con 0x7f878c13f710 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f878c1380f0 con 0x7f878c1430c0 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:20.220 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.218+0000 7f8790f3a640 10 monclient(hunting): authenticate will time out at 5616.364227s 2024-03-25T19:52:20.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.220+0000 7f878ad9d640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f878c1430c0 0x7f878c1e2eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:54558/0 (socket says 172.21.15.130:54558) 2024-03-25T19:52:20.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.220+0000 7f878ad9d640 1 -- 172.21.15.130:0/942786851 learned_addr learned my addr 172.21.15.130:0/942786851 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:20.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.221+0000 7f8782ffd640 10 client.?.objecter ms_handle_connect 0x7f878c1430c0 2024-03-25T19:52:20.221 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.221+0000 7f8782ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:20.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227025497 0 0) 0x7f878c1380f0 con 0x7f878c1430c0 2024-03-25T19:52:20.224 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:20.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:20.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:20.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 10 monclient(hunting): my global_id is 4691 2024-03-25T19:52:20.225 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.225+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f876c004eb0 con 0x7f878c1430c0 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2411513398 0 0) 0x7f876c004eb0 con 0x7f878c1430c0 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f878c1380f0 con 0x7f878c1430c0 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8784001fc0 con 0x7f878c1430c0 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.233+0000 7f8782ffd640 10 monclient(hunting): dump: 2024-03-25T19:52:20.233 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.234 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.234+0000 7f8782ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2601775307 0 0) 0x7f878c1380f0 con 0x7f878c1430c0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 >> v1:172.21.15.163:6789/0 conn(0x7f878c13f710 legacy=0x7f878c133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 >> v1:172.21.15.130:6789/0 conn(0x7f878c0a98f0 legacy=0x7f878c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 1 monclient: found mon.c 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f878c1397f0 con 0x7f878c1430c0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 10 monclient: _finish_auth 0 2024-03-25T19:52:20.240 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8782ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 5 monclient: authenticate success, global_id 4691 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 10 monclient: _renew_subs 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c1db280 con 0x7f878c1430c0 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 10 monclient: _renew_subs 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c133960 con 0x7f878c1430c0 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.239+0000 7f8790f3a640 1 librados: init done 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.240+0000 7f8790f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.240+0000 7f8790f3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.240+0000 7f8790f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.241 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.240+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c139a40 con 0x7f878c1430c0 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8784003c20 con 0x7f878c1430c0 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f8784004030 con 0x7f878c1430c0 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f8782ffd640 10 monclient: dump: 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.583 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.583+0000 7f878959a640 4 set_mon_vals no callback set 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.584+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f8784004250 con 0x7f878c1430c0 2024-03-25T19:52:20.584 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.584+0000 7f878a59c640 10 monclient: get_auth_request con 0x7f876c06dc50 auth_method 0 2024-03-25T19:52:20.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.585+0000 7f8782ffd640 10 client.4691.objecter ms_handle_connect 0x7f876c06dc50 2024-03-25T19:52:20.589 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f8784054cd0 con 0x7f878c1430c0 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 10 client.4691.objecter ms_dispatch 0x7f878c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 3 client.4691.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 3 client.4691.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f87840507e0 con 0x7f878c1430c0 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:20.590 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.590+0000 7f8782ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:20.849 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.850+0000 7f8790f3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:20.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.850+0000 7f8790f3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:20.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.850+0000 7f8790f3a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:20.850 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.850+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f878c137e10 con 0x7f878c1430c0 2024-03-25T19:52:20.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.879+0000 7f8782ffd640 1 -- 172.21.15.130:0/942786851 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f878c137e10 con 0x7f878c1430c0 2024-03-25T19:52:20.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.879+0000 7f8782ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:20.879 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.879+0000 7f8782ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:20.879 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:20.879 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:20.881 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 10 librados: watch_flush enter 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 10 librados: watch_flush exit 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 10 client.4691.objecter successfully canceled tick 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 >> v1:172.21.15.163:6800/1217380380 conn(0x7f876c06dc50 legacy=0x7f876c070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 10 monclient: shutdown 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 >> v1:172.21.15.130:6790/0 conn(0x7f878c1430c0 legacy=0x7f878c1e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 shutdown_connections 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.881+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 >> 172.21.15.130:0/942786851 conn(0x7f878c0b7ea0 msgr2=0x7f878c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.882+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 shutdown_connections 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.882+0000 7f8790f3a640 1 -- 172.21.15.130:0/942786851 wait complete. 2024-03-25T19:52:20.882 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:20.882+0000 7f8790f3a640 1 librados: shutdown 2024-03-25T19:52:20.895 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:27.702 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:27.822 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.822+0000 7f9a825f6640 10 monclient: get_monmap_and_config 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.822+0000 7f9a825f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.822+0000 7f9a825f6640 10 monclient: monmap: 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:27.823757+0000 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:27.823757+0000 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:27.823 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 1 Processor -- start 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 1 -- start start 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient: init 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c13f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:27.825 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c142520 con 0x7f9a7c13f710 2024-03-25T19:52:27.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0b28a0 con 0x7f9a7c1430c0 2024-03-25T19:52:27.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c0b2a30 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:27.826 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.824+0000 7f9a825f6640 10 monclient(hunting): authenticate will time out at 5623.970332s 2024-03-25T19:52:27.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.832+0000 7f9a7b7fe640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f9a7c13f710 0x7f9a7c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:51568/0 (socket says 172.21.15.130:51568) 2024-03-25T19:52:27.832 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.832+0000 7f9a7b7fe640 1 -- 172.21.15.130:0/1158673918 learned_addr learned my addr 172.21.15.130:0/1158673918 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:27.852 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001632513 0 0) 0x7f9a7c0b2a30 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): my global_id is 4694 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a64003520 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308275783 0 0) 0x7f9a7c142520 con 0x7f9a7c13f710 2024-03-25T19:52:27.853 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.853+0000 7f9a7affd640 10 monclient(hunting): my global_id is 4701 2024-03-25T19:52:27.854 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.854+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c0b2a30 con 0x7f9a7c13f710 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1768863831 0 0) 0x7f9a7c0b28a0 con 0x7f9a7c1430c0 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 10 monclient(hunting): my global_id is 4696 2024-03-25T19:52:27.865 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.865+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c142520 con 0x7f9a7c1430c0 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1485634742 0 0) 0x7f9a7c0b2a30 con 0x7f9a7c13f710 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 >> v1:172.21.15.130:6790/0 conn(0x7f9a7c0a98f0 legacy=0x7f9a7c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 >> v1:172.21.15.163:6789/0 conn(0x7f9a7c1430c0 legacy=0x7f9a7c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: _finish_hunting 0 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 monclient: found mon.noname-a 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c0b44c0 con 0x7f9a7c13f710 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: _finish_auth 0 2024-03-25T19:52:27.869 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: _check_auth_tickets 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a70001670 con 0x7f9a7c13f710 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a825f6640 5 monclient: authenticate success, global_id 4701 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a825f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a7affd640 10 monclient: dump: 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.870 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:27.871 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:27.871 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:27.871 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.871 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.869+0000 7f9a825f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:27.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a700017d0 con 0x7f9a7c13f710 2024-03-25T19:52:27.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:27.884 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 1 -- 172.21.15.130:0/1158673918 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a70001be0 con 0x7f9a7c13f710 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a7affd640 10 monclient: dump: 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a825f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:27.885 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.885+0000 7f9a825f6640 4 set_mon_vals no callback set 2024-03-25T19:52:27.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 10 monclient: shutdown 2024-03-25T19:52:27.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:27.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 1 -- 172.21.15.130:0/1158673918 >> v1:172.21.15.130:6789/0 conn(0x7f9a7c13f710 legacy=0x7f9a7c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:27.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 1 -- 172.21.15.130:0/1158673918 shutdown_connections 2024-03-25T19:52:27.887 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 1 -- 172.21.15.130:0/1158673918 >> 172.21.15.130:0/1158673918 conn(0x7f9a7c0b7ea0 msgr2=0x7f9a7c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 1 -- 172.21.15.130:0/1158673918 shutdown_connections 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.886+0000 7f9a825f6640 1 -- 172.21.15.130:0/1158673918 wait complete. 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.887+0000 7f9a825f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.887+0000 7f9a825f6640 10 monclient: monmap: 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.888 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.887+0000 7f9a825f6640 1 librados: starting msgr at 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.887+0000 7f9a825f6640 1 librados: starting objecter 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.887+0000 7f9a825f6640 1 Processor -- start 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.888+0000 7f9a825f6640 1 -- start start 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.888+0000 7f9a825f6640 1 librados: setting wanted keys 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.888+0000 7f9a825f6640 1 librados: calling monclient init 2024-03-25T19:52:27.889 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.888+0000 7f9a825f6640 10 monclient: init 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c139190 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c137590 con 0x7f9a7c1430c0 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7c138b40 con 0x7f9a7c13f710 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:27.890 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.889+0000 7f9a825f6640 10 monclient(hunting): authenticate will time out at 5624.034989s 2024-03-25T19:52:27.908 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.909+0000 7f9a7b7fe640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f9a7c13f710 0x7f9a7c1364e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:50976/0 (socket says 172.21.15.130:50976) 2024-03-25T19:52:27.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.909+0000 7f9a7b7fe640 1 -- 172.21.15.130:0/658208732 learned_addr learned my addr 172.21.15.130:0/658208732 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:27.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.910+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c13f710 2024-03-25T19:52:27.909 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.910+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:27.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.918+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c0a98f0 2024-03-25T19:52:27.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.918+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:27.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.920+0000 7f9a78ff9640 10 client.?.objecter ms_handle_connect 0x7f9a7c1430c0 2024-03-25T19:52:27.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.920+0000 7f9a78ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523916738 0 0) 0x7f9a7c138b40 con 0x7f9a7c13f710 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 10 monclient(hunting): my global_id is 4697 2024-03-25T19:52:27.921 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.921+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a68004eb0 con 0x7f9a7c13f710 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926601648 0 0) 0x7f9a7c139190 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 10 monclient(hunting): my global_id is 4704 2024-03-25T19:52:27.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.928+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7c138b40 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.936+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1433849624 0 0) 0x7f9a68004eb0 con 0x7f9a7c13f710 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.936+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a7c139190 con 0x7f9a7c13f710 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.936+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a70001c20 con 0x7f9a7c13f710 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.937+0000 7f9a78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.937+0000 7f9a78ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.937+0000 7f9a78ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:27.937 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.938 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.937+0000 7f9a78ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171439542 0 0) 0x7f9a7c137590 con 0x7f9a7c1430c0 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 10 monclient(hunting): my global_id is 4699 2024-03-25T19:52:27.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.939+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a68004eb0 con 0x7f9a7c1430c0 2024-03-25T19:52:27.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729562263 0 0) 0x7f9a7c138b40 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a7c137590 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a6c001cd0 con 0x7f9a7c0a98f0 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.959+0000 7f9a78ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:27.959 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:27.960 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:27.960 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.964 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 289834677 0 0) 0x7f9a68004eb0 con 0x7f9a7c1430c0 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a7c138b40 con 0x7f9a7c1430c0 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a74001850 con 0x7f9a7c1430c0 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.965+0000 7f9a78ff9640 10 monclient(hunting): dump: 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:27.965 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3043877004 0 0) 0x7f9a7c139190 con 0x7f9a7c13f710 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 >> v1:172.21.15.163:6789/0 conn(0x7f9a7c1430c0 legacy=0x7f9a7c1e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 >> v1:172.21.15.130:6789/0 conn(0x7f9a7c0a98f0 legacy=0x7f9a7c136e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 1 monclient: found mon.c 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a7c136bf0 con 0x7f9a7c13f710 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 10 monclient: _finish_auth 0 2024-03-25T19:52:27.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 5 monclient: authenticate success, global_id 4697 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 10 monclient: _renew_subs 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7c13afe0 con 0x7f9a7c13f710 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 10 monclient: _renew_subs 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a7c139620 con 0x7f9a7c13f710 2024-03-25T19:52:27.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.970+0000 7f9a825f6640 1 librados: init done 2024-03-25T19:52:27.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.971+0000 7f9a825f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:27.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.971+0000 7f9a825f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:27.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.971+0000 7f9a825f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:27.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:27.971+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7c074cc0 con 0x7f9a7c13f710 2024-03-25T19:52:28.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a70004a90 con 0x7f9a7c13f710 2024-03-25T19:52:28.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:28.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9a70004de0 con 0x7f9a7c13f710 2024-03-25T19:52:28.316 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a78ff9640 10 monclient: dump: 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.316+0000 7f9a7a7fc640 4 set_mon_vals no callback set 2024-03-25T19:52:28.317 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.317+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f9a70005000 con 0x7f9a7c13f710 2024-03-25T19:52:28.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.317+0000 7f9a7bfff640 10 monclient: get_auth_request con 0x7f9a6806e170 auth_method 0 2024-03-25T19:52:28.318 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.318+0000 7f9a78ff9640 10 client.4697.objecter ms_handle_connect 0x7f9a6806e170 2024-03-25T19:52:28.322 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f9a700549a0 con 0x7f9a7c13f710 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 10 client.4697.objecter ms_dispatch 0x7f9a7c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 3 client.4697.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 3 client.4697.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9a700504b0 con 0x7f9a7c13f710 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:28.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.323+0000 7f9a78ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:28.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.546+0000 7f9a825f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:28.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.546+0000 7f9a825f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:28.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.546+0000 7f9a825f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:28.546 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.546+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9a7c138f60 con 0x7f9a7c13f710 2024-03-25T19:52:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.582+0000 7f9a78ff9640 1 -- 172.21.15.130:0/658208732 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f9a700523e0 con 0x7f9a7c13f710 2024-03-25T19:52:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.582+0000 7f9a78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:28.582 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.582+0000 7f9a78ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:28.582 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:28.582 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.584+0000 7f9a825f6640 10 librados: watch_flush enter 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 10 librados: watch_flush exit 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 10 client.4697.objecter successfully canceled tick 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 >> v1:172.21.15.163:6800/1217380380 conn(0x7f9a6806e170 legacy=0x7f9a68070630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 10 monclient: shutdown 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:28.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 >> v1:172.21.15.130:6790/0 conn(0x7f9a7c13f710 legacy=0x7f9a7c1364e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 shutdown_connections 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 >> 172.21.15.130:0/658208732 conn(0x7f9a7c0b7ea0 msgr2=0x7f9a7c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 shutdown_connections 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 -- 172.21.15.130:0/658208732 wait complete. 2024-03-25T19:52:28.586 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:28.585+0000 7f9a825f6640 1 librados: shutdown 2024-03-25T19:52:28.596 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:31.007 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:15.940 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.012 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:15.945 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.039 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:15.972 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.058 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:15.991 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.064 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:15.965 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.071 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:15.972 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.073 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:16.006 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.091 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:15.991 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:31.106 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:16.040 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:35.409 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.532+0000 7f95ab00e640 10 monclient: get_monmap_and_config 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.532+0000 7f95ab00e640 10 monclient: build_initial_monmap 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.532+0000 7f95ab00e640 10 monclient: monmap: 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:35.533345+0000 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:35.533345+0000 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:35.532 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:35.533 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:35.533 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 1 Processor -- start 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 1 -- start start 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient: init 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-a con 0x7f95a40a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-c con 0x7f95a413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-b con 0x7f95a41430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a4142520 con 0x7f95a40a98f0 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a40b28a0 con 0x7f95a413f710 2024-03-25T19:52:35.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a40b2a30 con 0x7f95a41430c0 2024-03-25T19:52:35.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:35.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.533+0000 7f95ab00e640 10 monclient(hunting): authenticate will time out at 5631.679611s 2024-03-25T19:52:35.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.542+0000 7f95a8daa640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 0x7f95a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56482/0 (socket says 172.21.15.130:56482) 2024-03-25T19:52:35.542 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.542+0000 7f95a8daa640 1 -- 172.21.15.130:0/493020861 learned_addr learned my addr 172.21.15.130:0/493020861 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840871682 0 0) 0x7f95a4142520 con 0x7f95a40a98f0 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 10 monclient(hunting): _negotiate 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 10 monclient(hunting): my global_id is 4707 2024-03-25T19:52:35.549 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.549+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9594003520 con 0x7f95a40a98f0 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4038174102 0 0) 0x7f9594003520 con 0x7f95a40a98f0 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 >> v1:172.21.15.130:6790/0 conn(0x7f95a41430c0 legacy=0x7f95a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 >> v1:172.21.15.163:6789/0 conn(0x7f95a413f710 legacy=0x7f95a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: _finish_hunting 0 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 monclient: found mon.noname-a 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:35.562 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a40b44c0 con 0x7f95a40a98f0 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: _finish_auth 0 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: _check_auth_tickets 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9598001690 con 0x7f95a40a98f0 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95a37fe640 10 monclient: dump: 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:35.563 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:35.564 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:35.564 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:35.564 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95ab00e640 5 monclient: authenticate success, global_id 4707 2024-03-25T19:52:35.564 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.562+0000 7f95ab00e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95980017f0 con 0x7f95a40a98f0 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 1 -- 172.21.15.130:0/493020861 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9598001c00 con 0x7f95a40a98f0 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95a37fe640 10 monclient: dump: 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.573 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95ab00e640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:35.574 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.573+0000 7f95ab00e640 4 set_mon_vals no callback set 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 10 monclient: shutdown 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 1 -- 172.21.15.130:0/493020861 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 legacy=0x7f95a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 1 -- 172.21.15.130:0/493020861 shutdown_connections 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 1 -- 172.21.15.130:0/493020861 >> 172.21.15.130:0/493020861 conn(0x7f95a40b7ea0 msgr2=0x7f95a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 1 -- 172.21.15.130:0/493020861 shutdown_connections 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.574+0000 7f95ab00e640 1 -- 172.21.15.130:0/493020861 wait complete. 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 10 monclient: build_initial_monmap 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 10 monclient: monmap: 2024-03-25T19:52:35.576 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 librados: starting msgr at 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 librados: starting objecter 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 Processor -- start 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 -- start start 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 librados: setting wanted keys 2024-03-25T19:52:35.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 1 librados: calling monclient init 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.575+0000 7f95ab00e640 10 monclient: init 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-a con 0x7f95a40a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-b con 0x7f95a413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient(hunting): picked mon.noname-c con 0x7f95a41430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a4135ac0 con 0x7f95a40a98f0 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a4137030 con 0x7f95a413f710 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a41372a0 con 0x7f95a41430c0 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:35.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.576+0000 7f95ab00e640 10 monclient(hunting): authenticate will time out at 5631.722090s 2024-03-25T19:52:35.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.580+0000 7f95a8daa640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 0x7f95a4133440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:56508/0 (socket says 172.21.15.130:56508) 2024-03-25T19:52:35.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.580+0000 7f95a8daa640 1 -- 172.21.15.130:0/29135018 learned_addr learned my addr 172.21.15.130:0/29135018 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:35.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.581+0000 7f95a17fa640 10 client.?.objecter ms_handle_connect 0x7f95a40a98f0 2024-03-25T19:52:35.580 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.581+0000 7f95a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:35.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.585+0000 7f95a17fa640 10 client.?.objecter ms_handle_connect 0x7f95a413f710 2024-03-25T19:52:35.585 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.585+0000 7f95a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:35.591 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.592+0000 7f95a17fa640 10 client.?.objecter ms_handle_connect 0x7f95a41430c0 2024-03-25T19:52:35.592 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.592+0000 7f95a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:35.594 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944991495 0 0) 0x7f95a4135ac0 con 0x7f95a40a98f0 2024-03-25T19:52:35.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:52:35.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:35.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:35.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 10 monclient(hunting): my global_id is 4710 2024-03-25T19:52:35.595 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.595+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9588004eb0 con 0x7f95a40a98f0 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.604+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216183169 0 0) 0x7f95a41372a0 con 0x7f95a41430c0 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.605+0000 7f95a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.605+0000 7f95a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.605+0000 7f95a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.605+0000 7f95a17fa640 10 monclient(hunting): my global_id is 4703 2024-03-25T19:52:35.605 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.605+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a4135ac0 con 0x7f95a41430c0 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.607+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700300956 0 0) 0x7f95a4137030 con 0x7f95a413f710 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.607+0000 7f95a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.607+0000 7f95a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.607+0000 7f95a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.607+0000 7f95a17fa640 10 monclient(hunting): my global_id is 4705 2024-03-25T19:52:35.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.608+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a41372a0 con 0x7f95a413f710 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1748822901 0 0) 0x7f9588004eb0 con 0x7f95a40a98f0 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95a4137030 con 0x7f95a40a98f0 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9598001c00 con 0x7f95a40a98f0 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 10 monclient(hunting): dump: 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:35.609 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:35.610 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.613 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.609+0000 7f95a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:35.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2865554 0 0) 0x7f95a4137030 con 0x7f95a40a98f0 2024-03-25T19:52:35.614 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6790/0 conn(0x7f95a41430c0 legacy=0x7f95a41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.163:6789/0 conn(0x7f95a413f710 legacy=0x7f95a41379a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 1 monclient: found mon.a 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a4137580 con 0x7f95a40a98f0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 10 monclient: _finish_auth 0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.612+0000 7f95ab00e640 5 monclient: authenticate success, global_id 4710 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _renew_subs 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a41363b0 con 0x7f95a40a98f0 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 client.4710.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _renew_subs 2024-03-25T19:52:35.615 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a41353a0 con 0x7f95a40a98f0 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 1 librados: init done 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:35.616 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.613+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a40627b0 con 0x7f95a40a98f0 2024-03-25T19:52:35.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9598004a90 con 0x7f95a40a98f0 2024-03-25T19:52:35.926 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f9598004e20 con 0x7f95a40a98f0 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a17fa640 10 monclient: dump: 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:35.927 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:35.928 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:35.928 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:35.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.926+0000 7f95a2ffd640 4 set_mon_vals no callback set 2024-03-25T19:52:35.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.927+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f9598005040 con 0x7f95a40a98f0 2024-03-25T19:52:35.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.927+0000 7f95a3fff640 10 monclient: get_auth_request con 0x7f958806ddf0 auth_method 0 2024-03-25T19:52:35.929 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.928+0000 7f95a17fa640 10 client.4710.objecter ms_handle_connect 0x7f958806ddf0 2024-03-25T19:52:35.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f9598054940 con 0x7f95a40a98f0 2024-03-25T19:52:35.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 10 client.4710.objecter ms_dispatch 0x7f95a4000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:35.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 3 client.4710.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:35.932 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 3 client.4710.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:35.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 20 client.4710.objecter dump_active .. 0 homeless 2024-03-25T19:52:35.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9598050450 con 0x7f95a40a98f0 2024-03-25T19:52:35.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:35.933 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:35.932+0000 7f95a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:36.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95ab00e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:36.182 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95ab00e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95ab00e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95a413e1f0 con 0x7f95a40a98f0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a8daa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 legacy=0x7f95a4133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a8daa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 legacy=0x7f95a4133440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a8daa640 1 --1- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 0x7f95a4133440 :-1 s=OPENED pgs=80301 cs=1 l=1).handle_message read tag failed 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a8daa640 1 --1- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 0x7f95a4133440 :-1 s=OPENED pgs=80301 cs=1 l=1).fault on lossy channel, failing 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient: ms_handle_reset current mon v1:172.21.15.130:6789/0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f95a40a98f0 legacy=0x7f95a4133440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient(hunting): picked mon.a con 0x7f9588072e70 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient(hunting): picked mon.c con 0x7f95a41430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient(hunting): picked mon.b con 0x7f95a413f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95a4137030 con 0x7f9588072e70 2024-03-25T19:52:36.183 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f9588004eb0 con 0x7f95a413f710 2024-03-25T19:52:36.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f95880804e0 con 0x7f95a41430c0 2024-03-25T19:52:36.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.182+0000 7f95a17fa640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:36.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.183+0000 7f95a17fa640 10 client.4710.objecter ms_handle_connect 0x7f95a413f710 2024-03-25T19:52:36.184 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.183+0000 7f95a17fa640 10 client.4710.objecter resend_mon_ops 2024-03-25T19:52:36.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.189+0000 7f95a17fa640 10 client.4710.objecter ms_handle_connect 0x7f9588072e70 2024-03-25T19:52:36.188 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.189+0000 7f95a17fa640 10 client.4710.objecter resend_mon_ops 2024-03-25T19:52:36.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.248+0000 7f95a17fa640 10 client.4710.objecter ms_handle_connect 0x7f95a41430c0 2024-03-25T19:52:36.248 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.248+0000 7f95a17fa640 10 client.4710.objecter resend_mon_ops 2024-03-25T19:52:36.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.256+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698786035 0 0) 0x7f95880804e0 con 0x7f95a41430c0 2024-03-25T19:52:36.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.256+0000 7f95a17fa640 10 monclient(hunting): _negotiate 2024-03-25T19:52:36.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.256+0000 7f95a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:36.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.256+0000 7f95a17fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T19:52:36.256 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.256+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f958807e480 con 0x7f95a41430c0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 85314 0 0) 0x7f958807e480 con 0x7f95a41430c0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.163:6789/0 conn(0x7f95a413f710 legacy=0x7f958807c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6789/0 conn(0x7f9588072e70 legacy=0x7f9588075260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _finish_hunting 0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 monclient: found mon.c 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f9588081740 con 0x7f95a41430c0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95880804e0 con 0x7f95a41430c0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _finish_auth 0 2024-03-25T19:52:36.272 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: _check_auth_tickets 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f959c0019d0 con 0x7f95a41430c0 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.272+0000 7f95a17fa640 10 monclient: dump: 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:36.273 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:36.313 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.314+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f959c0017f0 con 0x7f95a41430c0 2024-03-25T19:52:36.313 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.314+0000 7f95a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:36.313 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.314+0000 7f95a2ffd640 4 set_mon_vals no callback set 2024-03-25T19:52:36.314 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.314+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f959c001bf0 con 0x7f95a41430c0 2024-03-25T19:52:36.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.320+0000 7f95a17fa640 1 -- 172.21.15.130:0/29135018 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f959c052ff0 con 0x7f95a41430c0 2024-03-25T19:52:36.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.320+0000 7f95a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:36.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.320+0000 7f95a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:36.320 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:36.320 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 10 librados: watch_flush enter 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 10 librados: watch_flush exit 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 20 client.4710.objecter shutdown clearing up homeless session... 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 10 client.4710.objecter successfully canceled tick 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.163:6800/1217380380 conn(0x7f958806ddf0 legacy=0x7f95880702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 10 monclient: shutdown 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.322+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 >> v1:172.21.15.130:6790/0 conn(0x7f95a41430c0 legacy=0x7f9588078cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.323+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 shutdown_connections 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.323+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 >> 172.21.15.130:0/29135018 conn(0x7f95a40b7ea0 msgr2=0x7f95a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.323+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 shutdown_connections 2024-03-25T19:52:36.323 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.323+0000 7f95ab00e640 1 -- 172.21.15.130:0/29135018 wait complete. 2024-03-25T19:52:36.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:36.323+0000 7f95ab00e640 1 librados: shutdown 2024-03-25T19:52:36.334 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:43.142 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.258+0000 7fd7bf5f6640 10 monclient: get_monmap_and_config 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.258+0000 7fd7bf5f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.258+0000 7fd7bf5f6640 10 monclient: monmap: 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:43.259570+0000 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:43.259570+0000 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:43.258 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:43.259 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:43.259 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:43.259 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.259+0000 7fd7bf5f6640 1 Processor -- start 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.259+0000 7fd7bf5f6640 1 -- start start 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.259+0000 7fd7bf5f6640 10 monclient: init 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b813f710 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b81430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b8142520 con 0x7fd7b813f710 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80b28a0 con 0x7fd7b81430c0 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80b2a30 con 0x7fd7b80a98f0 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:43.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.260+0000 7fd7bf5f6640 10 monclient(hunting): authenticate will time out at 5639.406228s 2024-03-25T19:52:43.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.271+0000 7fd7bd392640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fd7b80a98f0 0x7fd7b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:58368/0 (socket says 172.21.15.130:58368) 2024-03-25T19:52:43.271 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.271+0000 7fd7bd392640 1 -- 172.21.15.130:0/619382389 learned_addr learned my addr 172.21.15.130:0/619382389 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2902376968 0 0) 0x7fd7b80b2a30 con 0x7fd7b80a98f0 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 10 monclient(hunting): my global_id is 4706 2024-03-25T19:52:43.291 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.291+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a4003520 con 0x7fd7b80a98f0 2024-03-25T19:52:43.293 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707934059 0 0) 0x7fd7b8142520 con 0x7fd7b813f710 2024-03-25T19:52:43.293 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.293 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.293 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.293 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 10 monclient(hunting): my global_id is 4713 2024-03-25T19:52:43.294 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.294+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b80b2a30 con 0x7fd7b813f710 2024-03-25T19:52:43.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165646889 0 0) 0x7fd7b80b28a0 con 0x7fd7b81430c0 2024-03-25T19:52:43.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 10 monclient(hunting): my global_id is 4708 2024-03-25T19:52:43.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.301+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b8142520 con 0x7fd7b81430c0 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1601022636 0 0) 0x7fd7b80b2a30 con 0x7fd7b813f710 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 >> v1:172.21.15.130:6790/0 conn(0x7fd7b80a98f0 legacy=0x7fd7b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 >> v1:172.21.15.163:6789/0 conn(0x7fd7b81430c0 legacy=0x7fd7b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 monclient: found mon.noname-a 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b80b44c0 con 0x7fd7b813f710 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: _finish_auth 0 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd7a8001670 con 0x7fd7b813f710 2024-03-25T19:52:43.386 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7affff640 10 monclient: dump: 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7bf5f6640 5 monclient: authenticate success, global_id 4713 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.316+0000 7fd7bf5f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7a80017d0 con 0x7fd7b813f710 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:43.387 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 1 -- 172.21.15.130:0/619382389 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd7a8001be0 con 0x7fd7b813f710 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7bf5f6640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7bf5f6640 4 set_mon_vals no callback set 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.332+0000 7fd7affff640 10 monclient: dump: 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 10 monclient: shutdown 2024-03-25T19:52:43.388 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/619382389 >> v1:172.21.15.130:6789/0 conn(0x7fd7b813f710 legacy=0x7fd7b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/619382389 shutdown_connections 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/619382389 >> 172.21.15.130:0/619382389 conn(0x7fd7b80b7ea0 msgr2=0x7fd7b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/619382389 shutdown_connections 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.333+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/619382389 wait complete. 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.334+0000 7fd7bf5f6640 10 monclient: build_initial_monmap 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.334+0000 7fd7bf5f6640 10 monclient: monmap: 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:43.389 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.334+0000 7fd7bf5f6640 1 librados: starting msgr at 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.334+0000 7fd7bf5f6640 1 librados: starting objecter 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 Processor -- start 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 -- start start 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 librados: setting wanted keys 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 librados: calling monclient init 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient: init 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b80a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:43.390 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b81430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b81398e0 con 0x7fd7b80a98f0 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80ab8b0 con 0x7fd7b813f710 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7b80abb90 con 0x7fd7b81430c0 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.335+0000 7fd7bf5f6640 10 monclient(hunting): authenticate will time out at 5639.481418s 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.343+0000 7fd7bcb91640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fd7b813f710 0x7fd7b81df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:38926/0 (socket says 172.21.15.130:38926) 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.343+0000 7fd7bcb91640 1 -- 172.21.15.130:0/3965691671 learned_addr learned my addr 172.21.15.130:0/3965691671 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.344+0000 7fd7adffb640 10 client.?.objecter ms_handle_connect 0x7fd7b813f710 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.344+0000 7fd7adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.349+0000 7fd7adffb640 10 client.?.objecter ms_handle_connect 0x7fd7b81430c0 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.349+0000 7fd7adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223316069 0 0) 0x7fd7b80ab8b0 con 0x7fd7b813f710 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.391 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 10 monclient(hunting): my global_id is 4711 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.354+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd79c004eb0 con 0x7fd7b813f710 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.356+0000 7fd7adffb640 10 client.?.objecter ms_handle_connect 0x7fd7b80a98f0 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.356+0000 7fd7adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 341314681 0 0) 0x7fd79c004eb0 con 0x7fd7b813f710 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7b80ab8b0 con 0x7fd7b813f710 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd7a8001670 con 0x7fd7b813f710 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 10 monclient(hunting): dump: 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.392 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.361+0000 7fd7adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3070020805 0 0) 0x7fd7b80abb90 con 0x7fd7b81430c0 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 10 monclient(hunting): my global_id is 4709 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.365+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd79c004eb0 con 0x7fd7b81430c0 2024-03-25T19:52:43.393 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328266717 0 0) 0x7fd7b81398e0 con 0x7fd7b80a98f0 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 10 monclient(hunting): my global_id is 4716 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.370+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b80abb90 con 0x7fd7b80a98f0 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1823841283 0 0) 0x7fd7b80ab8b0 con 0x7fd7b813f710 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 >> v1:172.21.15.130:6790/0 conn(0x7fd7b81430c0 legacy=0x7fd7b81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 >> v1:172.21.15.130:6789/0 conn(0x7fd7b80a98f0 legacy=0x7fd7b8134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 1 monclient: found mon.b 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b8138a20 con 0x7fd7b813f710 2024-03-25T19:52:43.394 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 5 monclient: authenticate success, global_id 4711 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 10 monclient: _renew_subs 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7b8137840 con 0x7fd7b813f710 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 10 monclient: _renew_subs 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7b8138500 con 0x7fd7b813f710 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.384+0000 7fd7bf5f6640 1 librados: init done 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.385+0000 7fd7bf5f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.385+0000 7fd7bf5f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.385+0000 7fd7bf5f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:43.395 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.385+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7b80b27a0 con 0x7fd7b813f710 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7a80038a0 con 0x7fd7b813f710 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fd7a8003cf0 con 0x7fd7b813f710 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7adffb640 10 monclient: dump: 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:43.686 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.685+0000 7fd7af7fe640 4 set_mon_vals no callback set 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.686+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fd7a8003f10 con 0x7fd7b813f710 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.687+0000 7fd7bd392640 10 monclient: get_auth_request con 0x7fd79c06dea0 auth_method 0 2024-03-25T19:52:43.687 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.687+0000 7fd7adffb640 10 client.4711.objecter ms_handle_connect 0x7fd79c06dea0 2024-03-25T19:52:43.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7fd7a80549e0 con 0x7fd7b813f710 2024-03-25T19:52:43.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 10 client.4711.objecter ms_dispatch 0x7fd7b8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 3 client.4711.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 3 client.4711.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd7a80504f0 con 0x7fd7b813f710 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:43.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.691+0000 7fd7adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:43.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.939+0000 7fd7bf5f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:43.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.939+0000 7fd7bf5f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:43.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.939+0000 7fd7bf5f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:52:43.939 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.939+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7b80627b0 con 0x7fd7b813f710 2024-03-25T19:52:43.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.973+0000 7fd7adffb640 1 -- 172.21.15.130:0/3965691671 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7fd7a8052420 con 0x7fd7b813f710 2024-03-25T19:52:43.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.973+0000 7fd7adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:43.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.973+0000 7fd7adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:43.973 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:43.973 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.975+0000 7fd7bf5f6640 10 librados: watch_flush enter 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.975+0000 7fd7bf5f6640 10 librados: watch_flush exit 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.975+0000 7fd7bf5f6640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.975+0000 7fd7bf5f6640 10 client.4711.objecter successfully canceled tick 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 >> v1:172.21.15.163:6800/1217380380 conn(0x7fd79c06dea0 legacy=0x7fd79c070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 10 monclient: shutdown 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 >> v1:172.21.15.163:6789/0 conn(0x7fd7b813f710 legacy=0x7fd7b81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 shutdown_connections 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 >> 172.21.15.130:0/3965691671 conn(0x7fd7b80b7ea0 msgr2=0x7fd7b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 shutdown_connections 2024-03-25T19:52:43.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 -- 172.21.15.130:0/3965691671 wait complete. 2024-03-25T19:52:43.977 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:43.976+0000 7fd7bf5f6640 1 librados: shutdown 2024-03-25T19:52:43.989 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:50.796 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:50.914 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.912+0000 7f2053988640 10 monclient: get_monmap_and_config 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.912+0000 7f2053988640 10 monclient: build_initial_monmap 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.912+0000 7f2053988640 10 monclient: monmap: 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:50.912894+0000 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:50.912894+0000 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:50.915 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.912+0000 7f2053988640 1 Processor -- start 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 1 -- start start 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 10 monclient: init 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 10 monclient(hunting): picked mon.noname-c con 0x7f204c0a98f0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.913+0000 7f2053988640 10 monclient(hunting): picked mon.noname-b con 0x7f204c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 10 monclient(hunting): picked mon.noname-a con 0x7f204c1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c142520 con 0x7f204c1430c0 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c0b28a0 con 0x7f204c0a98f0 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c0b2a30 con 0x7f204c13f710 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:50.916 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.914+0000 7f2053988640 10 monclient(hunting): authenticate will time out at 5647.059661s 2024-03-25T19:52:50.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.918+0000 7f2051f25640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f204c1430c0 0x7f204c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43258/0 (socket says 172.21.15.130:43258) 2024-03-25T19:52:50.917 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.918+0000 7f2051f25640 1 -- 172.21.15.130:0/3303704279 learned_addr learned my addr 172.21.15.130:0/3303704279 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.930+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51591053 0 0) 0x7f204c142520 con 0x7f204c1430c0 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.931+0000 7f2043fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.931+0000 7f2043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.931+0000 7f2043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.931+0000 7f2043fff640 10 monclient(hunting): my global_id is 4719 2024-03-25T19:52:50.930 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.931+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2038003520 con 0x7f204c1430c0 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1115075873 0 0) 0x7f204c0b2a30 con 0x7f204c13f710 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 10 monclient(hunting): my global_id is 4712 2024-03-25T19:52:50.945 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.946+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f204c142520 con 0x7f204c13f710 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1071943185 0 0) 0x7f2038003520 con 0x7f204c1430c0 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 >> v1:172.21.15.130:6790/0 conn(0x7f204c13f710 legacy=0x7f204c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 >> v1:172.21.15.163:6789/0 conn(0x7f204c0a98f0 legacy=0x7f204c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 monclient: found mon.noname-a 2024-03-25T19:52:50.948 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.130:6789/0 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c0b44c0 con 0x7f204c1430c0 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: _finish_auth 0 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2044001a10 con 0x7f204c1430c0 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2053988640 5 monclient: authenticate success, global_id 4719 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2053988640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2043fff640 10 monclient: dump: 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:50.949 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:50.950 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.948+0000 7f2053988640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:50.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.0 v1:172.21.15.130:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2044001b70 con 0x7f204c1430c0 2024-03-25T19:52:50.968 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 1 -- 172.21.15.130:0/3303704279 <== mon.0 v1:172.21.15.130:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2044001f80 con 0x7f204c1430c0 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2053988640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2053988640 4 set_mon_vals no callback set 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.968+0000 7f2043fff640 10 monclient: dump: 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:50.969 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:50.970 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:50.970 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:50.970 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:50.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.969+0000 7f2053988640 10 monclient: shutdown 2024-03-25T19:52:50.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.969+0000 7f2053988640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.969+0000 7f2053988640 1 -- 172.21.15.130:0/3303704279 >> v1:172.21.15.130:6789/0 conn(0x7f204c1430c0 legacy=0x7f204c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 1 -- 172.21.15.130:0/3303704279 shutdown_connections 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 1 -- 172.21.15.130:0/3303704279 >> 172.21.15.130:0/3303704279 conn(0x7f204c0b7ea0 msgr2=0x7f204c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 1 -- 172.21.15.130:0/3303704279 shutdown_connections 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 1 -- 172.21.15.130:0/3303704279 wait complete. 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 10 monclient: build_initial_monmap 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.970+0000 7f2053988640 10 monclient: monmap: 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:50.972 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 librados: starting msgr at 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 librados: starting objecter 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 Processor -- start 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 -- start start 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 librados: setting wanted keys 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 librados: calling monclient init 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient: init 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:52:50.973 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient(hunting): picked mon.noname-a con 0x7f204c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient(hunting): picked mon.noname-b con 0x7f204c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient(hunting): picked mon.noname-c con 0x7f204c1430c0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c0aba30 con 0x7f204c0a98f0 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c1d7810 con 0x7f204c13f710 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204c1d8440 con 0x7f204c1430c0 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:50.974 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.971+0000 7f2053988640 10 monclient(hunting): authenticate will time out at 5647.117599s 2024-03-25T19:52:50.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.985+0000 7f2051724640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f204c0a98f0 0x7f204c1d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:43290/0 (socket says 172.21.15.130:43290) 2024-03-25T19:52:50.984 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.985+0000 7f2051724640 1 -- 172.21.15.130:0/2466349873 learned_addr learned my addr 172.21.15.130:0/2466349873 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:50.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.986+0000 7f2041ffb640 10 client.?.objecter ms_handle_connect 0x7f204c0a98f0 2024-03-25T19:52:50.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.986+0000 7f2041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:50.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.987+0000 7f2041ffb640 10 client.?.objecter ms_handle_connect 0x7f204c13f710 2024-03-25T19:52:50.986 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.987+0000 7f2041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:50.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.989+0000 7f2041ffb640 10 client.?.objecter ms_handle_connect 0x7f204c1430c0 2024-03-25T19:52:50.988 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.989+0000 7f2041ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209322400 0 0) 0x7f204c0aba30 con 0x7f204c0a98f0 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 10 monclient(hunting): my global_id is 4722 2024-03-25T19:52:50.997 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:50.997+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030004eb0 con 0x7f204c0a98f0 2024-03-25T19:52:51.004 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962437840 0 0) 0x7f204c1d8440 con 0x7f204c1430c0 2024-03-25T19:52:51.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:51.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:51.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:51.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 10 monclient(hunting): my global_id is 4715 2024-03-25T19:52:51.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.005+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f204c0aba30 con 0x7f204c1430c0 2024-03-25T19:52:51.052 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:35.985 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.063 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:35.996 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3407717076 0 0) 0x7f204c0aba30 con 0x7f204c1430c0 2024-03-25T19:52:51.070 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f204c1d8440 con 0x7f204c1430c0 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2044001e80 con 0x7f204c1430c0 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 10 monclient(hunting): dump: 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:51.071 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:51.072 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:51.072 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.071+0000 7f2041ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:51.098 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:36.031 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56156742 0 0) 0x7f204c1d8440 con 0x7f204c1430c0 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 >> v1:172.21.15.163:6789/0 conn(0x7f204c13f710 legacy=0x7f204c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 >> v1:172.21.15.130:6789/0 conn(0x7f204c0a98f0 legacy=0x7f204c1d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 1 monclient: found mon.c 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:51.101 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204c138d60 con 0x7f204c1430c0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 5 monclient: authenticate success, global_id 4715 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 10 monclient: _renew_subs 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204c138f50 con 0x7f204c1430c0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 10 client.4715.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 10 monclient: _renew_subs 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f204c1d51e0 con 0x7f204c1430c0 2024-03-25T19:52:51.102 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.101+0000 7f2053988640 1 librados: init done 2024-03-25T19:52:51.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.102+0000 7f2053988640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:51.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.102+0000 7f2053988640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:51.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.102+0000 7f2053988640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:51.103 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.102+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204c0627b0 con 0x7f204c1430c0 2024-03-25T19:52:51.114 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:36.048 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.116 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:36.017 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.120 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:36.021 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.130 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:36.063 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.151 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:36.052 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.159 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:36.092 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20440015c0 con 0x7f204c1430c0 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f2044004e10 con 0x7f204c1430c0 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f2041ffb640 10 monclient: dump: 2024-03-25T19:52:51.407 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:51.408 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.407+0000 7f20437fe640 4 set_mon_vals no callback set 2024-03-25T19:52:51.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.408+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f204401ade0 con 0x7f204c1430c0 2024-03-25T19:52:51.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.409+0000 7f2051724640 10 monclient: get_auth_request con 0x7f203006ddd0 auth_method 0 2024-03-25T19:52:51.409 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.409+0000 7f2041ffb640 10 client.4715.objecter ms_handle_connect 0x7f203006ddd0 2024-03-25T19:52:51.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f2044054a40 con 0x7f204c1430c0 2024-03-25T19:52:51.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 10 client.4715.objecter ms_dispatch 0x7f204c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:51.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 3 client.4715.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:51.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 3 client.4715.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:51.413 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 20 client.4715.objecter dump_active .. 0 homeless 2024-03-25T19:52:51.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2044050380 con 0x7f204c1430c0 2024-03-25T19:52:51.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:51.414 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.414+0000 7f2041ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:51.618 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.619+0000 7f2053988640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:51.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.619+0000 7f2053988640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:51.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.619+0000 7f2053988640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:52:51.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.619+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f204c13e250 con 0x7f204c1430c0 2024-03-25T19:52:51.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.666+0000 7f2041ffb640 1 -- 172.21.15.130:0/2466349873 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f2044052480 con 0x7f204c1430c0 2024-03-25T19:52:51.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.666+0000 7f2041ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:51.666 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.666+0000 7f2041ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:51.667 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:51.667 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.668+0000 7f2053988640 10 librados: watch_flush enter 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 10 librados: watch_flush exit 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 20 client.4715.objecter shutdown clearing up homeless session... 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 10 client.4715.objecter successfully canceled tick 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 >> v1:172.21.15.163:6800/1217380380 conn(0x7f203006ddd0 legacy=0x7f2030070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 10 monclient: shutdown 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 >> v1:172.21.15.130:6790/0 conn(0x7f204c1430c0 legacy=0x7f204c138170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 shutdown_connections 2024-03-25T19:52:51.669 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 >> 172.21.15.130:0/2466349873 conn(0x7f204c0b7ea0 msgr2=0x7f204c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:51.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 shutdown_connections 2024-03-25T19:52:51.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 -- 172.21.15.130:0/2466349873 wait complete. 2024-03-25T19:52:51.670 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:51.669+0000 7f2053988640 1 librados: shutdown 2024-03-25T19:52:51.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:52:58.487 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.603+0000 7fc1a3443640 10 monclient: get_monmap_and_config 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.603+0000 7fc1a3443640 10 monclient: build_initial_monmap 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.603+0000 7fc1a3443640 10 monclient: monmap: 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:52:58.605205+0000 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:52:58.605205+0000 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:52:58.604 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:52:58.605 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 1 Processor -- start 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 1 -- start start 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient: init 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c0a98f0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c13f710 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c1430c0 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.604+0000 7fc1a3443640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c142520 con 0x7fc19c0a98f0 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.605+0000 7fc1a3443640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b28a0 con 0x7fc19c1430c0 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.605+0000 7fc1a3443640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0b2a30 con 0x7fc19c13f710 2024-03-25T19:52:58.606 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.605+0000 7fc1a3443640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:58.607 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.605+0000 7fc1a3443640 10 monclient(hunting): authenticate will time out at 5654.751643s 2024-03-25T19:52:58.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.618+0000 7fc1a11df640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fc19c0a98f0 0x7fc19c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:47424/0 (socket says 172.21.15.130:47424) 2024-03-25T19:52:58.619 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.618+0000 7fc1a11df640 1 -- 172.21.15.130:0/3042296171 learned_addr learned my addr 172.21.15.130:0/3042296171 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 226165428 0 0) 0x7fc19c0b28a0 con 0x7fc19c1430c0 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 10 monclient(hunting): my global_id is 4720 2024-03-25T19:52:58.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.643+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc188003520 con 0x7fc19c1430c0 2024-03-25T19:52:58.650 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536400779 0 0) 0x7fc19c142520 con 0x7fc19c0a98f0 2024-03-25T19:52:58.650 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.650 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.650 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 10 monclient(hunting): my global_id is 4725 2024-03-25T19:52:58.651 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.650+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c0b28a0 con 0x7fc19c0a98f0 2024-03-25T19:52:58.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735265896 0 0) 0x7fc19c0b2a30 con 0x7fc19c13f710 2024-03-25T19:52:58.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.657 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 10 monclient(hunting): my global_id is 4718 2024-03-25T19:52:58.658 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.657+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c142520 con 0x7fc19c13f710 2024-03-25T19:52:58.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 37432792 0 0) 0x7fc188003520 con 0x7fc19c1430c0 2024-03-25T19:52:58.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 >> v1:172.21.15.130:6790/0 conn(0x7fc19c13f710 legacy=0x7fc19c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:58.660 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 >> v1:172.21.15.130:6789/0 conn(0x7fc19c0a98f0 legacy=0x7fc19c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: _finish_hunting 0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 monclient: found mon.noname-c 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0b44c0 con 0x7fc19c1430c0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: _finish_auth 0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: _check_auth_tickets 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc194001a10 con 0x7fc19c1430c0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc193fff640 10 monclient: dump: 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:58.661 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc1a3443640 5 monclient: authenticate success, global_id 4720 2024-03-25T19:52:58.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.660+0000 7fc1a3443640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc194001b70 con 0x7fc19c1430c0 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 1 -- 172.21.15.130:0/3042296171 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc194001f80 con 0x7fc19c1430c0 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc1a3443640 10 monclient: get_monmap_and_config success 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc1a3443640 4 set_mon_vals no callback set 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.685+0000 7fc193fff640 10 monclient: dump: 2024-03-25T19:52:58.686 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:58.687 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:58.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 10 monclient: shutdown 2024-03-25T19:52:58.688 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 1 -- 172.21.15.130:0/3042296171 >> v1:172.21.15.163:6789/0 conn(0x7fc19c1430c0 legacy=0x7fc19c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 1 -- 172.21.15.130:0/3042296171 shutdown_connections 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 1 -- 172.21.15.130:0/3042296171 >> 172.21.15.130:0/3042296171 conn(0x7fc19c0b7ea0 msgr2=0x7fc19c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.686+0000 7fc1a3443640 1 -- 172.21.15.130:0/3042296171 shutdown_connections 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.687+0000 7fc1a3443640 1 -- 172.21.15.130:0/3042296171 wait complete. 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.687+0000 7fc1a3443640 10 monclient: build_initial_monmap 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.687+0000 7fc1a3443640 10 monclient: monmap: 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:52:58.689 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 librados: starting msgr at 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 librados: starting objecter 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 Processor -- start 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 -- start start 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 librados: setting wanted keys 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 1 librados: calling monclient init 2024-03-25T19:52:58.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 10 monclient: init 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 10 monclient: _reopen_session rank -1 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.688+0000 7fc1a3443640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c0a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c13f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c1430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:52:58.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1d7500 con 0x7fc19c1430c0 2024-03-25T19:52:58.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1394a0 con 0x7fc19c13f710 2024-03-25T19:52:58.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c138b50 con 0x7fc19c0a98f0 2024-03-25T19:52:58.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 10 monclient(hunting): _renew_subs 2024-03-25T19:52:58.692 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.689+0000 7fc1a3443640 10 monclient(hunting): authenticate will time out at 5654.835687s 2024-03-25T19:52:58.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.701+0000 7fc1a19e0640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7fc19c1430c0 0x7fc19c134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:47466/0 (socket says 172.21.15.130:47466) 2024-03-25T19:52:58.702 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.701+0000 7fc1a19e0640 1 -- 172.21.15.130:0/3616190036 learned_addr learned my addr 172.21.15.130:0/3616190036 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:52:58.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.702+0000 7fc191ffb640 10 client.?.objecter ms_handle_connect 0x7fc19c1430c0 2024-03-25T19:52:58.703 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.702+0000 7fc191ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:58.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.705+0000 7fc191ffb640 10 client.?.objecter ms_handle_connect 0x7fc19c13f710 2024-03-25T19:52:58.705 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.705+0000 7fc191ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:58.710 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.709+0000 7fc191ffb640 10 client.?.objecter ms_handle_connect 0x7fc19c0a98f0 2024-03-25T19:52:58.710 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.709+0000 7fc191ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3766287775 0 0) 0x7fc19c1d7500 con 0x7fc19c1430c0 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 10 monclient(hunting): my global_id is 4728 2024-03-25T19:52:58.717 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.716+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc180004eb0 con 0x7fc19c1430c0 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792519197 0 0) 0x7fc19c1394a0 con 0x7fc19c13f710 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 10 monclient(hunting): my global_id is 4723 2024-03-25T19:52:58.721 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.720+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c1d7500 con 0x7fc19c13f710 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292944841 0 0) 0x7fc19c138b50 con 0x7fc19c0a98f0 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 10 monclient(hunting): _negotiate 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 10 monclient(hunting): my global_id is 4721 2024-03-25T19:52:58.727 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.726+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c1394a0 con 0x7fc19c0a98f0 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.735+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1296756488 0 0) 0x7fc180004eb0 con 0x7fc19c1430c0 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.735+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c138b50 con 0x7fc19c1430c0 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.736+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc194001ec0 con 0x7fc19c1430c0 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.736+0000 7fc191ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.736+0000 7fc191ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.736+0000 7fc191ffb640 10 monclient(hunting): dump: 2024-03-25T19:52:58.736 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:58.737 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.736+0000 7fc191ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 628721895 0 0) 0x7fc19c138b50 con 0x7fc19c1430c0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 >> v1:172.21.15.130:6790/0 conn(0x7fc19c0a98f0 legacy=0x7fc19c138440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 >> v1:172.21.15.163:6789/0 conn(0x7fc19c13f710 legacy=0x7fc19c1d7c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 10 monclient: _finish_hunting 0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 1 monclient: found mon.a 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c1d8380 con 0x7fc19c1430c0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 10 monclient: _finish_auth 0 2024-03-25T19:52:58.745 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 10 monclient: _check_auth_tickets 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc191ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc1a3443640 5 monclient: authenticate success, global_id 4728 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.743+0000 7fc1a3443640 10 monclient: _renew_subs 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c139710 con 0x7fc19c1430c0 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 10 client.4728.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 10 monclient: _renew_subs 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1d7750 con 0x7fc19c1430c0 2024-03-25T19:52:58.746 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.744+0000 7fc1a3443640 1 librados: init done 2024-03-25T19:52:58.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.745+0000 7fc1a3443640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:58.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.745+0000 7fc1a3443640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:58.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.745+0000 7fc1a3443640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:58.747 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:58.745+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc19c0627b0 con 0x7fc19c1430c0 2024-03-25T19:52:59.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc194003c20 con 0x7fc19c1430c0 2024-03-25T19:52:59.052 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fc194002100 con 0x7fc19c1430c0 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc191ffb640 10 monclient: dump: 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:52:59.053 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.051+0000 7fc1937fe640 4 set_mon_vals no callback set 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.052+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fc1940040f0 con 0x7fc19c1430c0 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.053+0000 7fc1a11df640 10 monclient: get_auth_request con 0x7fc18006ddf0 auth_method 0 2024-03-25T19:52:59.054 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.053+0000 7fc191ffb640 10 client.4728.objecter ms_handle_connect 0x7fc18006ddf0 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7fc194054b40 con 0x7fc19c1430c0 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 10 client.4728.objecter ms_dispatch 0x7fc19c000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 3 client.4728.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 3 client.4728.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 20 client.4728.objecter dump_active .. 0 homeless 2024-03-25T19:52:59.059 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc194050650 con 0x7fc19c1430c0 2024-03-25T19:52:59.060 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:52:59.060 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.058+0000 7fc191ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:52:59.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.284+0000 7fc1a3443640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:59.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.284+0000 7fc1a3443640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:59.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.284+0000 7fc1a3443640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:52:59.285 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.284+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc19c13fdc0 con 0x7fc19c1430c0 2024-03-25T19:52:59.320 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.320+0000 7fc191ffb640 1 -- 172.21.15.130:0/3616190036 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7fc194052580 con 0x7fc19c1430c0 2024-03-25T19:52:59.321 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.320+0000 7fc191ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:52:59.321 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.320+0000 7fc191ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:52:59.321 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:52:59.321 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.322+0000 7fc1a3443640 10 librados: watch_flush enter 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.322+0000 7fc1a3443640 10 librados: watch_flush exit 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 20 client.4728.objecter shutdown clearing up homeless session... 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 10 client.4728.objecter successfully canceled tick 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 >> v1:172.21.15.163:6800/1217380380 conn(0x7fc18006ddf0 legacy=0x7fc1800702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 10 monclient: shutdown 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:52:59.324 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 >> v1:172.21.15.130:6789/0 conn(0x7fc19c1430c0 legacy=0x7fc19c134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:52:59.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 shutdown_connections 2024-03-25T19:52:59.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 >> 172.21.15.130:0/3616190036 conn(0x7fc19c0b7ea0 msgr2=0x7fc19c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:52:59.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 shutdown_connections 2024-03-25T19:52:59.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 -- 172.21.15.130:0/3616190036 wait complete. 2024-03-25T19:52:59.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:52:59.323+0000 7fc1a3443640 1 librados: shutdown 2024-03-25T19:52:59.337 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:03.580 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:48.513 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.588 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:48.521 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.590 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:48.523 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.593 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:48.526 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.595 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:48.496 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.597 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:48.498 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.610 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:48.543 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.613 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:48.546 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:03.615 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:48.516 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:06.144 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:06.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.258+0000 7ffae2754640 10 monclient: get_monmap_and_config 2024-03-25T19:53:06.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.258+0000 7ffae2754640 10 monclient: build_initial_monmap 2024-03-25T19:53:06.258 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.258+0000 7ffae2754640 10 monclient: monmap: 2024-03-25T19:53:06.258 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:06.258 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:53:06.259679+0000 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:53:06.259679+0000 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:53:06.259 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.259+0000 7ffae2754640 1 Processor -- start 2024-03-25T19:53:06.260 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.259+0000 7ffae2754640 1 -- start start 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.259+0000 7ffae2754640 10 monclient: init 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-a con 0x7ffadc0b4d60 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-b con 0x7ffadc0b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-c con 0x7ffadc14be90 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc0af940 con 0x7ffadc0b4d60 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc0b0a70 con 0x7ffadc14be90 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc0b0d50 con 0x7ffadc0b1730 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:06.261 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.260+0000 7ffae2754640 10 monclient(hunting): authenticate will time out at 5662.406375s 2024-03-25T19:53:06.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.269+0000 7ffae0cf1640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7ffadc14be90 0x7ffadc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:60804/0 (socket says 172.21.15.130:60804) 2024-03-25T19:53:06.269 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.269+0000 7ffae0cf1640 1 -- 172.21.15.130:0/2992651633 learned_addr learned my addr 172.21.15.130:0/2992651633 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036881703 0 0) 0x7ffadc0b0a70 con 0x7ffadc14be90 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 10 monclient(hunting): my global_id is 4726 2024-03-25T19:53:06.280 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.280+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac4003520 con 0x7ffadc14be90 2024-03-25T19:53:06.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1452754157 0 0) 0x7ffac4003520 con 0x7ffadc14be90 2024-03-25T19:53:06.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 >> v1:172.21.15.130:6790/0 conn(0x7ffadc0b1730 legacy=0x7ffadc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 >> v1:172.21.15.130:6789/0 conn(0x7ffadc0b4d60 legacy=0x7ffadc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: _finish_hunting 0 2024-03-25T19:53:06.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 monclient: found mon.noname-c 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.163:6789/0 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc0b1030 con 0x7ffadc14be90 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: _finish_auth 0 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: _check_auth_tickets 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffad4001a10 con 0x7ffadc14be90 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffadaffd640 10 monclient: dump: 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.298 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffae2754640 5 monclient: authenticate success, global_id 4726 2024-03-25T19:53:06.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.297+0000 7ffae2754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 <== mon.1 v1:172.21.15.163:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad4001b70 con 0x7ffadc14be90 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 1 -- 172.21.15.130:0/2992651633 <== mon.1 v1:172.21.15.163:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffad4001f80 con 0x7ffadc14be90 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffadaffd640 10 monclient: dump: 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:06.325 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffae2754640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:06.326 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.325+0000 7ffae2754640 4 set_mon_vals no callback set 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 10 monclient: shutdown 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 1 -- 172.21.15.130:0/2992651633 >> v1:172.21.15.163:6789/0 conn(0x7ffadc14be90 legacy=0x7ffadc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 1 -- 172.21.15.130:0/2992651633 shutdown_connections 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 1 -- 172.21.15.130:0/2992651633 >> 172.21.15.130:0/2992651633 conn(0x7ffadc0617a0 msgr2=0x7ffadc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:06.328 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 1 -- 172.21.15.130:0/2992651633 shutdown_connections 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.326+0000 7ffae2754640 1 -- 172.21.15.130:0/2992651633 wait complete. 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 10 monclient: build_initial_monmap 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 10 monclient: monmap: 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 librados: starting msgr at 2024-03-25T19:53:06.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 librados: starting objecter 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 Processor -- start 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 -- start start 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 librados: setting wanted keys 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 1 librados: calling monclient init 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.327+0000 7ffae2754640 10 monclient: init 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-a con 0x7ffadc0b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-b con 0x7ffadc0b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient(hunting): picked mon.noname-c con 0x7ffadc14be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc1366f0 con 0x7ffadc0b1730 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc137fc0 con 0x7ffadc0b4d60 2024-03-25T19:53:06.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffadc139070 con 0x7ffadc14be90 2024-03-25T19:53:06.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:06.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.328+0000 7ffae2754640 10 monclient(hunting): authenticate will time out at 5662.473913s 2024-03-25T19:53:06.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.342+0000 7ffadbfff640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7ffadc0b1730 0x7ffadc12fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:52142/0 (socket says 172.21.15.130:52142) 2024-03-25T19:53:06.341 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.342+0000 7ffadbfff640 1 -- 172.21.15.130:0/747401725 learned_addr learned my addr 172.21.15.130:0/747401725 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:06.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.343+0000 7ffad8ff9640 10 client.?.objecter ms_handle_connect 0x7ffadc0b1730 2024-03-25T19:53:06.342 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.343+0000 7ffad8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:06.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.349+0000 7ffad8ff9640 10 client.?.objecter ms_handle_connect 0x7ffadc14be90 2024-03-25T19:53:06.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.349+0000 7ffad8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:06.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.350+0000 7ffad8ff9640 10 client.?.objecter ms_handle_connect 0x7ffadc0b4d60 2024-03-25T19:53:06.349 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.350+0000 7ffad8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3044431209 0 0) 0x7ffadc1366f0 con 0x7ffadc0b1730 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 10 monclient(hunting): my global_id is 4734 2024-03-25T19:53:06.351 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.351+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac8004eb0 con 0x7ffadc0b1730 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239694764 0 0) 0x7ffadc139070 con 0x7ffadc14be90 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 10 monclient(hunting): my global_id is 4727 2024-03-25T19:53:06.356 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.356+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffadc1366f0 con 0x7ffadc14be90 2024-03-25T19:53:06.361 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3717454266 0 0) 0x7ffac8004eb0 con 0x7ffadc0b1730 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffadc139070 con 0x7ffadc0b1730 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffacc001a30 con 0x7ffadc0b1730 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 10 monclient(hunting): dump: 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:06.362 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:06.363 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:06.363 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.363 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.362+0000 7ffad8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1365521351 0 0) 0x7ffadc1366f0 con 0x7ffadc14be90 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffac8004eb0 con 0x7ffadc14be90 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffad4001e80 con 0x7ffadc14be90 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:06.371 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.371+0000 7ffad8ff9640 10 monclient(hunting): dump: 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:06.372 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107556348 0 0) 0x7ffadc137fc0 con 0x7ffadc0b4d60 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 10 monclient(hunting): _negotiate 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 10 monclient(hunting): my global_id is 4729 2024-03-25T19:53:06.373 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.372+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffadc1366f0 con 0x7ffadc0b4d60 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2277385511 0 0) 0x7ffadc139070 con 0x7ffadc0b1730 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 >> v1:172.21.15.130:6790/0 conn(0x7ffadc14be90 legacy=0x7ffadc144210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 >> v1:172.21.15.163:6789/0 conn(0x7ffadc0b4d60 legacy=0x7ffadc143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 10 monclient: _finish_hunting 0 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 1 monclient: found mon.a 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:06.379 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffadc135f20 con 0x7ffadc0b1730 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 10 monclient: _finish_auth 0 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 10 monclient: _check_auth_tickets 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffad8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 5 monclient: authenticate success, global_id 4734 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 10 monclient: _renew_subs 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffadc136110 con 0x7ffadc0b1730 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 10 client.4734.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 10 monclient: _renew_subs 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffadc130650 con 0x7ffadc0b1730 2024-03-25T19:53:06.380 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.379+0000 7ffae2754640 1 librados: init done 2024-03-25T19:53:06.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.380+0000 7ffae2754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:06.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.380+0000 7ffae2754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:06.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.380+0000 7ffae2754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:53:06.381 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.380+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffadc0627b0 con 0x7ffadc0b1730 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffacc001c70 con 0x7ffadc0b1730 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7ffacc003a60 con 0x7ffadc0b1730 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffad8ff9640 10 monclient: dump: 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:06.689 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:06.690 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.689+0000 7ffada7fc640 4 set_mon_vals no callback set 2024-03-25T19:53:06.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.690+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7ffacc003c40 con 0x7ffadc0b1730 2024-03-25T19:53:06.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.691+0000 7ffadb7fe640 10 monclient: get_auth_request con 0x7ffac806df40 auth_method 0 2024-03-25T19:53:06.691 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.691+0000 7ffad8ff9640 10 client.4734.objecter ms_handle_connect 0x7ffac806df40 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7ffacc0548b0 con 0x7ffadc0b1730 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 10 client.4734.objecter ms_dispatch 0x7ffadc000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 3 client.4734.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 3 client.4734.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 20 client.4734.objecter dump_active .. 0 homeless 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffacc0503c0 con 0x7ffadc0b1730 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:06.695 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.695+0000 7ffad8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:06.943 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.944+0000 7ffae2754640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:06.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.944+0000 7ffae2754640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:06.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.944+0000 7ffae2754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.130:6789/0 2024-03-25T19:53:06.944 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.944+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 --> v1:172.21.15.130:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffadc072730 con 0x7ffadc0b1730 2024-03-25T19:53:06.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.978+0000 7ffad8ff9640 1 -- 172.21.15.130:0/747401725 <== mon.0 v1:172.21.15.130:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7ffacc0522f0 con 0x7ffadc0b1730 2024-03-25T19:53:06.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.978+0000 7ffad8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:06.978 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.978+0000 7ffad8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:06.978 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:53:06.978 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 10 librados: watch_flush enter 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 10 librados: watch_flush exit 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 20 client.4734.objecter shutdown clearing up homeless session... 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 10 client.4734.objecter successfully canceled tick 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 >> v1:172.21.15.163:6800/1217380380 conn(0x7ffac806df40 legacy=0x7ffac8070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 10 monclient: shutdown 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.980+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 >> v1:172.21.15.130:6789/0 conn(0x7ffadc0b1730 legacy=0x7ffadc12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.981+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 shutdown_connections 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.981+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 >> 172.21.15.130:0/747401725 conn(0x7ffadc0617a0 msgr2=0x7ffadc13c250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.981+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 shutdown_connections 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.981+0000 7ffae2754640 1 -- 172.21.15.130:0/747401725 wait complete. 2024-03-25T19:53:06.981 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:06.981+0000 7ffae2754640 1 librados: shutdown 2024-03-25T19:53:06.993 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:11.118 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:56.050 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.122 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:56.055 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.145 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:11:56.078 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.170 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:56.071 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.173 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:56.106 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.178 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:56.111 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.179 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:56.081 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.206 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:11:56.107 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:11.215 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:11:56.148 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:13.799 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.913+0000 7f95ff23e640 10 monclient: get_monmap_and_config 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.913+0000 7f95ff23e640 10 monclient: build_initial_monmap 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.913+0000 7f95ff23e640 10 monclient: monmap: 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:53:13.914504+0000 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:53:13.914504+0000 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:53:13.913 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.914+0000 7f95ff23e640 1 Processor -- start 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.914+0000 7f95ff23e640 1 -- start start 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.914+0000 7f95ff23e640 10 monclient: init 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-b con 0x7f95f80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-c con 0x7f95f813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-a con 0x7f95f81430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f8142520 con 0x7f95f81430c0 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f80b28a0 con 0x7f95f813f710 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f80b2a30 con 0x7f95f80a98f0 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:13.915 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.915+0000 7f95ff23e640 10 monclient(hunting): authenticate will time out at 5670.060885s 2024-03-25T19:53:13.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.919+0000 7f95fcfda640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7f95f80a98f0 0x7f95f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:60122/0 (socket says 172.21.15.130:60122) 2024-03-25T19:53:13.919 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.919+0000 7f95fcfda640 1 -- 172.21.15.130:0/4218125145 learned_addr learned my addr 172.21.15.130:0/4218125145 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:13.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2561580792 0 0) 0x7f95f80b2a30 con 0x7f95f80a98f0 2024-03-25T19:53:13.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 10 monclient(hunting): _negotiate 2024-03-25T19:53:13.927 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:13.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:13.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 10 monclient(hunting): my global_id is 4730 2024-03-25T19:53:13.928 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.928+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e0003520 con 0x7f95f80a98f0 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2896397614 0 0) 0x7f95e0003520 con 0x7f95f80a98f0 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 >> v1:172.21.15.163:6789/0 conn(0x7f95f813f710 legacy=0x7f95f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 >> v1:172.21.15.130:6789/0 conn(0x7f95f81430c0 legacy=0x7f95f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: _finish_hunting 0 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 monclient: found mon.noname-b 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:53:13.934 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f80b44c0 con 0x7f95f80a98f0 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: _finish_auth 0 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: _check_auth_tickets 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ff23e640 5 monclient: authenticate success, global_id 4730 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95f4001690 con 0x7f95f80a98f0 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ef7fe640 10 monclient: dump: 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:13.935 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:13.936 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:13.936 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:13.936 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:13.936 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.934+0000 7f95ff23e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95f40017f0 con 0x7f95f80a98f0 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 1 -- 172.21.15.130:0/4218125145 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95f4001c00 con 0x7f95f80a98f0 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ef7fe640 10 monclient: dump: 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:13.953 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:13.954 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:13.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ff23e640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:13.954 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.953+0000 7f95ff23e640 4 set_mon_vals no callback set 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 10 monclient: shutdown 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 1 -- 172.21.15.130:0/4218125145 >> v1:172.21.15.130:6790/0 conn(0x7f95f80a98f0 legacy=0x7f95f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 1 -- 172.21.15.130:0/4218125145 shutdown_connections 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 1 -- 172.21.15.130:0/4218125145 >> 172.21.15.130:0/4218125145 conn(0x7f95f80b7ea0 msgr2=0x7f95f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:13.955 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 1 -- 172.21.15.130:0/4218125145 shutdown_connections 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.954+0000 7f95ff23e640 1 -- 172.21.15.130:0/4218125145 wait complete. 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.955+0000 7f95ff23e640 10 monclient: build_initial_monmap 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.955+0000 7f95ff23e640 10 monclient: monmap: 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 librados: starting msgr at 2024-03-25T19:53:13.956 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 librados: starting objecter 2024-03-25T19:53:13.957 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 Processor -- start 2024-03-25T19:53:13.957 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 -- start start 2024-03-25T19:53:13.957 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 librados: setting wanted keys 2024-03-25T19:53:13.957 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 1 librados: calling monclient init 2024-03-25T19:53:13.957 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 10 monclient: init 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.956+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-c con 0x7f95f80a98f0 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-b con 0x7f95f813f710 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 10 monclient(hunting): picked mon.noname-a con 0x7f95f81430c0 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f8135730 con 0x7f95f81430c0 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f81358c0 con 0x7f95f813f710 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95f8135a50 con 0x7f95f80a98f0 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:13.958 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.957+0000 7f95ff23e640 10 monclient(hunting): authenticate will time out at 5670.102834s 2024-03-25T19:53:13.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.972+0000 7f95fd7db640 1 --1- >> v1:172.21.15.130:6789/0 conn(0x7f95f81430c0 0x7f95f81d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.130:6789/0 says I am v1:172.21.15.130:60138/0 (socket says 172.21.15.130:60138) 2024-03-25T19:53:13.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.972+0000 7f95fd7db640 1 -- 172.21.15.130:0/4251760534 learned_addr learned my addr 172.21.15.130:0/4251760534 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:13.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.973+0000 7f95ed7fa640 10 client.?.objecter ms_handle_connect 0x7f95f81430c0 2024-03-25T19:53:13.972 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.973+0000 7f95ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:13.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.977+0000 7f95ed7fa640 10 client.?.objecter ms_handle_connect 0x7f95f813f710 2024-03-25T19:53:13.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.977+0000 7f95ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:13.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.981+0000 7f95ed7fa640 10 client.?.objecter ms_handle_connect 0x7f95f80a98f0 2024-03-25T19:53:13.980 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.981+0000 7f95ed7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:13.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236144415 0 0) 0x7f95f8135730 con 0x7f95f81430c0 2024-03-25T19:53:13.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:53:13.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:13.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:13.990 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 10 monclient(hunting): my global_id is 4740 2024-03-25T19:53:13.991 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.991+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95e4004eb0 con 0x7f95f81430c0 2024-03-25T19:53:13.993 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488495301 0 0) 0x7f95f81358c0 con 0x7f95f813f710 2024-03-25T19:53:13.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:53:13.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:13.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:13.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 10 monclient(hunting): my global_id is 4735 2024-03-25T19:53:13.994 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:13.994+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f8135730 con 0x7f95f813f710 2024-03-25T19:53:14.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 107350529 0 0) 0x7f95f8135a50 con 0x7f95f80a98f0 2024-03-25T19:53:14.005 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 10 monclient(hunting): _negotiate 2024-03-25T19:53:14.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:14.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:14.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 10 monclient(hunting): my global_id is 4733 2024-03-25T19:53:14.006 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.006+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95f81358c0 con 0x7f95f80a98f0 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659447902 0 0) 0x7f95f8135730 con 0x7f95f813f710 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f8135a50 con 0x7f95f813f710 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95e80018a0 con 0x7f95f813f710 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 10 monclient(hunting): dump: 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.017 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:14.018 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.017+0000 7f95ed7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.0 v1:172.21.15.130:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3135773393 0 0) 0x7f95e4004eb0 con 0x7f95f81430c0 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.130:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95f8135730 con 0x7f95f81430c0 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.0 v1:172.21.15.130:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95f0001c90 con 0x7f95f81430c0 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.019+0000 7f95ed7fa640 10 monclient(hunting): dump: 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:14.019 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:14.020 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354634040 0 0) 0x7f95f81358c0 con 0x7f95f80a98f0 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95e4004eb0 con 0x7f95f80a98f0 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95f4001b00 con 0x7f95f80a98f0 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.031+0000 7f95ed7fa640 10 monclient(hunting): dump: 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:14.031 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:14.032 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 166062100 0 0) 0x7f95f8135a50 con 0x7f95f813f710 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 >> v1:172.21.15.130:6790/0 conn(0x7f95f80a98f0 legacy=0x7f95f81dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 >> v1:172.21.15.130:6789/0 conn(0x7f95f81430c0 legacy=0x7f95f81d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 10 monclient: _finish_hunting 0 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 1 monclient: found mon.b 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95f81d5000 con 0x7f95f813f710 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 10 monclient: _finish_auth 0 2024-03-25T19:53:14.039 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 10 monclient: _check_auth_tickets 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 5 monclient: authenticate success, global_id 4735 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 10 monclient: _renew_subs 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95f81d52b0 con 0x7f95f813f710 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 10 client.4735.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 10 monclient: _renew_subs 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95f813c900 con 0x7f95f813f710 2024-03-25T19:53:14.040 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.039+0000 7f95ff23e640 1 librados: init done 2024-03-25T19:53:14.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.040+0000 7f95ff23e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:14.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.040+0000 7f95ff23e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:14.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.040+0000 7f95ff23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:53:14.041 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.040+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95f813cbb0 con 0x7f95f813f710 2024-03-25T19:53:14.329 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95e8001d50 con 0x7f95f813f710 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7f95e8003630 con 0x7f95f813f710 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95ed7fa640 10 monclient: dump: 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:14.330 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.330+0000 7f95eeffd640 4 set_mon_vals no callback set 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.331+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7f95e8019640 con 0x7f95f813f710 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.331+0000 7f95fcfda640 10 monclient: get_auth_request con 0x7f95e406e060 auth_method 0 2024-03-25T19:53:14.331 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.332+0000 7f95ed7fa640 10 client.4735.objecter ms_handle_connect 0x7f95e406e060 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7f95e8053420 con 0x7f95f813f710 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 10 client.4735.objecter ms_dispatch 0x7f95f8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 3 client.4735.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 3 client.4735.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 20 client.4735.objecter dump_active .. 0 homeless 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f95e804ed60 con 0x7f95f813f710 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:14.336 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.336+0000 7f95ed7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:14.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.560+0000 7f95ff23e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:14.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.560+0000 7f95ff23e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:14.559 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.560+0000 7f95ff23e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.163:6789/0 2024-03-25T19:53:14.560 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.560+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 --> v1:172.21.15.163:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95f81371c0 con 0x7f95f813f710 2024-03-25T19:53:14.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.599+0000 7f95ed7fa640 1 -- 172.21.15.130:0/4251760534 <== mon.1 v1:172.21.15.163:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7f95e8050e60 con 0x7f95f813f710 2024-03-25T19:53:14.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.599+0000 7f95ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:14.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.599+0000 7f95ed7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:14.599 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:53:14.599 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 10 librados: watch_flush enter 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 10 librados: watch_flush exit 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 20 client.4735.objecter shutdown clearing up homeless session... 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 10 client.4735.objecter successfully canceled tick 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 >> v1:172.21.15.163:6800/1217380380 conn(0x7f95e406e060 legacy=0x7f95e4070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 10 monclient: shutdown 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.601+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 >> v1:172.21.15.163:6789/0 conn(0x7f95f813f710 legacy=0x7f95f81d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.602+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 shutdown_connections 2024-03-25T19:53:14.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.602+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 >> 172.21.15.130:0/4251760534 conn(0x7f95f80b7ea0 msgr2=0x7f95f814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:14.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.602+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 shutdown_connections 2024-03-25T19:53:14.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.602+0000 7f95ff23e640 1 -- 172.21.15.130:0/4251760534 wait complete. 2024-03-25T19:53:14.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:14.602+0000 7f95ff23e640 1 librados: shutdown 2024-03-25T19:53:14.614 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:21.420 DEBUG:teuthology.orchestra.run.smithi130:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T19:53:21.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.533+0000 7fb5bce95640 10 monclient: get_monmap_and_config 2024-03-25T19:53:21.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.533+0000 7fb5bce95640 10 monclient: build_initial_monmap 2024-03-25T19:53:21.532 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.533+0000 7fb5bce95640 10 monclient: monmap: 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:53:21.534203+0000 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:53:21.534203+0000 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-c 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-b 2024-03-25T19:53:21.533 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.534+0000 7fb5bce95640 1 Processor -- start 2024-03-25T19:53:21.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.534+0000 7fb5bce95640 1 -- start start 2024-03-25T19:53:21.534 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.534+0000 7fb5bce95640 10 monclient: init 2024-03-25T19:53:21.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:21.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T19:53:21.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b80b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:21.535 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b80b1730 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-a con 0x7fb5b814be90 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b80af940 con 0x7fb5b814be90 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b80b0a70 con 0x7fb5b80b4d60 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b80b0d50 con 0x7fb5b80b1730 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:21.536 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.535+0000 7fb5bce95640 10 monclient(hunting): authenticate will time out at 5677.681063s 2024-03-25T19:53:21.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.539+0000 7fb5b5d9b640 1 --1- >> v1:172.21.15.130:6790/0 conn(0x7fb5b80b1730 0x7fb5b80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.130:6790/0 says I am v1:172.21.15.130:51692/0 (socket says 172.21.15.130:51692) 2024-03-25T19:53:21.538 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.539+0000 7fb5b5d9b640 1 -- 172.21.15.130:0/2791375804 learned_addr learned my addr 172.21.15.130:0/2791375804 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797657488 0 0) 0x7fb5b80b0d50 con 0x7fb5b80b1730 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 10 monclient(hunting): my global_id is 4736 2024-03-25T19:53:21.558 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.558+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5a0003520 con 0x7fb5b80b1730 2024-03-25T19:53:21.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248017457 0 0) 0x7fb5b80af940 con 0x7fb5b814be90 2024-03-25T19:53:21.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.568 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 10 monclient(hunting): my global_id is 4743 2024-03-25T19:53:21.569 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.569+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b80b0d50 con 0x7fb5b814be90 2024-03-25T19:53:21.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2580929404 0 0) 0x7fb5b80b0a70 con 0x7fb5b80b4d60 2024-03-25T19:53:21.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.575 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 10 monclient(hunting): my global_id is 4738 2024-03-25T19:53:21.576 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.576+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b80af940 con 0x7fb5b80b4d60 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2185766491 0 0) 0x7fb5a0003520 con 0x7fb5b80b1730 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 >> v1:172.21.15.163:6789/0 conn(0x7fb5b80b4d60 legacy=0x7fb5b813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 >> v1:172.21.15.130:6789/0 conn(0x7fb5b814be90 legacy=0x7fb5b814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: _finish_hunting 0 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 monclient: found mon.noname-b 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.130:6790/0 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b80b1030 con 0x7fb5b80b1730 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: _finish_auth 0 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: _check_auth_tickets 2024-03-25T19:53:21.577 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb5ac001690 con 0x7fb5b80b1730 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5bce95640 5 monclient: authenticate success, global_id 4736 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5bce95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5b559a640 10 monclient: dump: 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:21.578 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.579 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:21.579 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:21.579 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:21.579 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.579 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.577+0000 7fb5bce95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.2 v1:172.21.15.130:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5ac0017f0 con 0x7fb5b80b1730 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 1 -- 172.21.15.130:0/2791375804 <== mon.2 v1:172.21.15.130:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb5ac001c00 con 0x7fb5b80b1730 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5b559a640 10 monclient: dump: 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.597 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5bce95640 10 monclient: get_monmap_and_config success 2024-03-25T19:53:21.598 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.597+0000 7fb5bce95640 4 set_mon_vals no callback set 2024-03-25T19:53:21.601 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 10 monclient: shutdown 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 1 -- 172.21.15.130:0/2791375804 >> v1:172.21.15.130:6790/0 conn(0x7fb5b80b1730 legacy=0x7fb5b80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 1 -- 172.21.15.130:0/2791375804 shutdown_connections 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 1 -- 172.21.15.130:0/2791375804 >> 172.21.15.130:0/2791375804 conn(0x7fb5b80617a0 msgr2=0x7fb5b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 1 -- 172.21.15.130:0/2791375804 shutdown_connections 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.598+0000 7fb5bce95640 1 -- 172.21.15.130:0/2791375804 wait complete. 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.599+0000 7fb5bce95640 10 monclient: build_initial_monmap 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.599+0000 7fb5bce95640 10 monclient: monmap: 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:epoch 0 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:last_changed 0.000000 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:created 0.000000 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 0 (unknown) 2024-03-25T19:53:21.602 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 1 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.noname-a 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.noname-b 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.noname-c 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 librados: starting msgr at 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 librados: starting objecter 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 Processor -- start 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 -- start start 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 librados: setting wanted keys 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 1 librados: calling monclient init 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.600+0000 7fb5bce95640 10 monclient: init 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient: _reopen_session rank -1 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-a con 0x7fb5b80b1730 addr v1:172.21.15.130:6789/0 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-b con 0x7fb5b80b4d60 addr v1:172.21.15.163:6789/0 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient(hunting): picked mon.noname-c con 0x7fb5b814be90 addr v1:172.21.15.130:6790/0 2024-03-25T19:53:21.603 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 1 -- --> v1:172.21.15.130:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b8137d90 con 0x7fb5b80b1730 2024-03-25T19:53:21.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 1 -- --> v1:172.21.15.163:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b8142110 con 0x7fb5b80b4d60 2024-03-25T19:53:21.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 1 -- --> v1:172.21.15.130:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5b8140b20 con 0x7fb5b814be90 2024-03-25T19:53:21.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient(hunting): _renew_subs 2024-03-25T19:53:21.604 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.601+0000 7fb5bce95640 10 monclient(hunting): authenticate will time out at 5677.746852s 2024-03-25T19:53:21.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.608+0000 7fb5b5d9b640 1 --1- >> v1:172.21.15.163:6789/0 conn(0x7fb5b80b4d60 0x7fb5b8133fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.163:6789/0 says I am v1:172.21.15.130:37592/0 (socket says 172.21.15.130:37592) 2024-03-25T19:53:21.608 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.608+0000 7fb5b5d9b640 1 -- 172.21.15.130:0/3707455791 learned_addr learned my addr 172.21.15.130:0/3707455791 (peer_addr_for_me v1:172.21.15.130:0/0) 2024-03-25T19:53:21.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.612+0000 7fb5a6ffd640 10 client.?.objecter ms_handle_connect 0x7fb5b80b4d60 2024-03-25T19:53:21.612 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.612+0000 7fb5a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:21.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.1 v1:172.21.15.163:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4093921705 0 0) 0x7fb5b8142110 con 0x7fb5b80b4d60 2024-03-25T19:53:21.625 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 10 monclient(hunting): my global_id is 4741 2024-03-25T19:53:21.626 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.626+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.163:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598004eb0 con 0x7fb5b80b4d60 2024-03-25T19:53:21.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.629+0000 7fb5a6ffd640 10 client.?.objecter ms_handle_connect 0x7fb5b814be90 2024-03-25T19:53:21.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.629+0000 7fb5a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:21.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.629+0000 7fb5a6ffd640 10 client.?.objecter ms_handle_connect 0x7fb5b80b1730 2024-03-25T19:53:21.629 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.629+0000 7fb5a6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T19:53:21.640 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2025308004 0 0) 0x7fb5b8140b20 con 0x7fb5b814be90 2024-03-25T19:53:21.640 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 10 monclient(hunting): my global_id is 4739 2024-03-25T19:53:21.641 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.641+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5b8142110 con 0x7fb5b814be90 2024-03-25T19:53:21.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.1 v1:172.21.15.163:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2893607289 0 0) 0x7fb598004eb0 con 0x7fb5b80b4d60 2024-03-25T19:53:21.642 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.163:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5b8140b20 con 0x7fb5b80b4d60 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.1 v1:172.21.15.163:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb5ac001bc0 con 0x7fb5b80b4d60 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:21.643 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.644 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:21.644 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:21.644 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:21.644 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.644 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.643+0000 7fb5a6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.0 v1:172.21.15.130:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1017205259 0 0) 0x7fb5b8137d90 con 0x7fb5b80b1730 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 10 monclient(hunting): my global_id is 4746 2024-03-25T19:53:21.646 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.646+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb598004eb0 con 0x7fb5b80b1730 2024-03-25T19:53:21.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.653+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3205332154 0 0) 0x7fb5b8142110 con 0x7fb5b814be90 2024-03-25T19:53:21.652 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.653+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5b8137d90 con 0x7fb5b814be90 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb5b0001c10 con 0x7fb5b814be90 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 10 monclient(hunting): dump: 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:21.661 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2284321778 0 0) 0x7fb5b8137d90 con 0x7fb5b814be90 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 >> v1:172.21.15.163:6789/0 conn(0x7fb5b80b4d60 legacy=0x7fb5b8133fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 >> v1:172.21.15.130:6789/0 conn(0x7fb5b80b1730 legacy=0x7fb5b8140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 monclient: found mon.c 2024-03-25T19:53:21.662 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.660+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5b8140cb0 con 0x7fb5b814be90 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5a6ffd640 10 monclient: _finish_auth 0 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 5 monclient: authenticate success, global_id 4739 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 10 monclient: _renew_subs 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5b8140ef0 con 0x7fb5b814be90 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 10 client.4739.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 10 monclient: _renew_subs 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5b8142710 con 0x7fb5b814be90 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.661+0000 7fb5bce95640 1 librados: init done 2024-03-25T19:53:21.663 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.662+0000 7fb5bce95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:21.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.662+0000 7fb5bce95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:21.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.662+0000 7fb5bce95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:53:21.664 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.662+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5b80b2270 con 0x7fb5b814be90 2024-03-25T19:53:21.969 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.969+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5b00020a0 con 0x7fb5b814be90 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.969+0000 7fb5a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.969+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2603185530 0 0) 0x7fb5b0003b10 con 0x7fb5b814be90 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.969+0000 7fb5a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.970+0000 7fb5a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.970+0000 7fb5a6ffd640 10 monclient: dump: 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:epoch 1 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:fsid 2da1777d-8d35-40d5-acd9-30de4511aae0 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:last_changed 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:created 2024-03-25T19:40:42.193886+0000 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:min_mon_release 19 (squid) 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:election_strategy: 3 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:0: v1:172.21.15.130:6789/0 mon.a 2024-03-25T19:53:21.970 INFO:teuthology.misc.health.smithi130.stderr:1: v1:172.21.15.163:6789/0 mon.b 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr:2: v1:172.21.15.130:6790/0 mon.c 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr: 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.970+0000 7fb5b4d99640 4 set_mon_vals no callback set 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.970+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 4238262918 0 0) 0x7fb5b0019b20 con 0x7fb5b814be90 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.971+0000 7fb5b659c640 10 monclient: get_auth_request con 0x7fb59806dee0 auth_method 0 2024-03-25T19:53:21.971 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.971+0000 7fb5a6ffd640 10 client.4739.objecter ms_handle_connect 0x7fb59806dee0 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 1645+0+0 (unknown 2680401452 0 0) 0x7fb5b0053800 con 0x7fb5b814be90 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 10 client.4739.objecter ms_dispatch 0x7fb5b8000b90 osd_map(12..12 src has 1..12) v4 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 3 client.4739.objecter handle_osd_map got epochs [12,12] > 0 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 3 client.4739.objecter handle_osd_map decoding full epoch 12 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 20 client.4739.objecter dump_active .. 0 homeless 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb5b004f140 con 0x7fb5b814be90 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T19:53:21.976 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:21.976+0000 7fb5a6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T19:53:22.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.219+0000 7fb5bce95640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:22.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.219+0000 7fb5bce95640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:22.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.219+0000 7fb5bce95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.130:6790/0 2024-03-25T19:53:22.219 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.219+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 --> v1:172.21.15.130:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb5b80627b0 con 0x7fb5b814be90 2024-03-25T19:53:22.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.297+0000 7fb5a6ffd640 1 -- 172.21.15.130:0/3707455791 <== mon.2 v1:172.21.15.130:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7100 (unknown 4250787923 0 2273107309) 0x7fb5b0051240 con 0x7fb5b814be90 2024-03-25T19:53:22.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.297+0000 7fb5a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T19:53:22.297 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.297+0000 7fb5a6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T19:53:22.297 INFO:teuthology.misc.health.smithi130.stdout: 2024-03-25T19:53:22.297 INFO:teuthology.misc.health.smithi130.stdout:{"epoch":12,"fsid":"2da1777d-8d35-40d5-acd9-30de4511aae0","created":"2024-03-25T19:41:42.362589+0000","modified":"2024-03-25T19:51:50.784492+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T19:51:50.784492+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"874157ee-ff52-4d50-bd9d-aa9258173515","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"d9cf0e41-1e91-411e-bf6c-05195eff515a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"c216a970-462b-4157-8e38-8f989b551f16","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"543e2e52-08be-4e00-9c0d-105ff80c5d7f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fcbf4be-140e-4f80-aa30-f7799e4db30b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ef685213-8889-4279-b9f3-5fb85b5936f7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"36fb90b8-9c7b-4ac1-a2b2-0004fd9541de","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"8cce7c73-66f0-4ffc-93c3-bde0072d6f30","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 10 librados: watch_flush enter 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 10 librados: watch_flush exit 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 20 client.4739.objecter shutdown clearing up homeless session... 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 10 client.4739.objecter successfully canceled tick 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 >> v1:172.21.15.163:6800/1217380380 conn(0x7fb59806dee0 legacy=0x7fb5980703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 10 monclient: shutdown 2024-03-25T19:53:22.299 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 >> v1:172.21.15.130:6790/0 conn(0x7fb5b814be90 legacy=0x7fb5b81e3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 shutdown_connections 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 >> 172.21.15.130:0/3707455791 conn(0x7fb5b80617a0 msgr2=0x7fb5b813b770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 shutdown_connections 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 -- 172.21.15.130:0/3707455791 wait complete. 2024-03-25T19:53:22.300 INFO:teuthology.misc.health.smithi130.stderr:2024-03-25T19:53:22.299+0000 7fb5bce95640 1 librados: shutdown 2024-03-25T19:53:22.308 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T19:53:22.308 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T19:53:22.310 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T19:53:22.310 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T19:53:22.310 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:24.225 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:11:35.964 35287== 2024-03-25T19:53:24.225 INFO:tasks.ceph.osd.0.smithi130.stderr:==00:00:11:35.964 35287== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:24.264 INFO:tasks.ceph.osd.0.smithi130.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:28.412 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:28.413 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T19:53:28.413 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T19:53:28.414 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T19:53:28.414 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:30.307 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:11:42.041 35292== 2024-03-25T19:53:30.307 INFO:tasks.ceph.osd.1.smithi130.stderr:==00:00:11:42.041 35292== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:30.352 INFO:tasks.ceph.osd.1.smithi130.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:31.241 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:16.173 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.249 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:16.182 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.275 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:16.208 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.293 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:16.226 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.301 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:16.202 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.302 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:16.236 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.312 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:16.213 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.334 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:16.267 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:31.334 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:16.235 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:34.516 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T19:53:34.516 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T19:53:34.517 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T19:53:34.517 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:36.376 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:11:48.110 35291== 2024-03-25T19:53:36.376 INFO:tasks.ceph.osd.2.smithi130.stderr:==00:00:11:48.110 35291== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:36.413 INFO:tasks.ceph.osd.2.smithi130.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:40.618 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T19:53:40.618 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T19:53:40.619 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T19:53:40.619 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:42.390 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:11:54.129 35288== 2024-03-25T19:53:42.391 INFO:tasks.ceph.osd.3.smithi130.stderr:==00:00:11:54.129 35288== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:42.442 INFO:tasks.ceph.osd.3.smithi130.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:46.721 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi130 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T19:53:46.721 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T19:53:46.721 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T19:53:46.721 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:48.672 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:12:00.395 35424== 2024-03-25T19:53:48.672 INFO:tasks.ceph.osd.4.smithi163.stderr:==00:00:12:00.395 35424== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:48.717 INFO:tasks.ceph.osd.4.smithi163.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:51.230 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:36.163 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.235 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:36.168 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.257 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:12:36.191 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.281 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:36.182 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.284 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:36.217 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.286 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:36.187 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.286 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:36.220 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.306 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:12:36.239 34972== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:51.310 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:12:36.211 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:53:52.823 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:52.823 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T19:53:52.824 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T19:53:52.824 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T19:53:52.824 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:53:54.663 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:12:06.384 35425== 2024-03-25T19:53:54.663 INFO:tasks.ceph.osd.5.smithi163.stderr:==00:00:12:06.384 35425== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:53:54.704 INFO:tasks.ceph.osd.5.smithi163.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:53:58.926 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:53:58.926 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T19:53:58.927 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T19:53:58.927 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T19:53:58.927 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:00.584 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:12:12.294 35429== 2024-03-25T19:54:00.584 INFO:tasks.ceph.osd.6.smithi163.stderr:==00:00:12:12.294 35429== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:54:00.631 INFO:tasks.ceph.osd.6.smithi163.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:54:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:54:05.029 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T19:54:05.029 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T19:54:05.030 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T19:54:05.030 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:06.598 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:12:18.309 35426== 2024-03-25T19:54:06.599 INFO:tasks.ceph.osd.7.smithi163.stderr:==00:00:12:18.309 35426== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:54:06.635 INFO:tasks.ceph.osd.7.smithi163.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:54:11.132 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T19:54:11.132 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi163 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T19:54:11.132 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T19:54:11.132 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T19:54:11.133 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T19:54:11.133 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:11.159 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T19:54:11.159 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T19:54:11.159 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:11.188 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T19:54:11.188 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T19:54:11.189 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T19:54:11.189 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T19:54:11.220 INFO:tasks.ceph.mon.a.smithi130.stderr:2024-03-25T19:54:11.205+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34945) UID: 0 2024-03-25T19:54:11.220 INFO:tasks.ceph.mon.a.smithi130.stderr:2024-03-25T19:54:11.207+0000 9f86640 -1 mon.a@0(peon) e1 *** Got Signal Terminated *** 2024-03-25T19:54:26.313 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:13:11.245 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:26.315 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:13:11.249 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:26.327 INFO:tasks.ceph.mon.c.smithi130.stderr:==00:00:13:11.260 34975== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:26.368 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:13:11.269 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:26.370 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:13:11.271 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:54:26.381 INFO:tasks.ceph.mon.b.smithi163.stderr:==00:00:13:11.282 34827== Warning: unimplemented fcntl command: 1036 2024-03-25T19:59:10.262 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T19:59:10.263 DEBUG:teuthology.orchestra.run.smithi130:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T19:59:10.306 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:59:10.311 DEBUG:teuthology.orchestra.run.smithi130:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T19:59:10.437 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:59:10.437 DEBUG:teuthology.orchestra.run.smithi130:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T19:59:10.502 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:59:10.502 DEBUG:teuthology.orchestra.run.smithi130:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T19:59:10.561 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:59:10.562 DEBUG:teuthology.orchestra.run.smithi130:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T19:59:10.646 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:59:10.646 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T19:59:10.724 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:59:10.725 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T19:59:10.790 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:59:10.790 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T19:59:10.831 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:17:55.764 34972== 2024-03-25T19:59:10.831 INFO:tasks.ceph.mon.a.smithi130.stderr:==00:00:17:55.764 34972== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T19:59:10.840 INFO:tasks.ceph.mon.a.smithi130.stderr:daemon-helper: command failed with exit status 42 2024-03-25T19:59:10.848 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:59:10.849 DEBUG:teuthology.orchestra.run.smithi163:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T19:59:10.905 INFO:tasks.ceph:Archiving mon data... 2024-03-25T19:59:10.906 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/data/mon.a.tgz 2024-03-25T19:59:10.907 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-03-25T19:59:10.922 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.EFqwodJ63b 2024-03-25T19:59:10.926 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.EFqwodJ63b 2024-03-25T19:59:11.018 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.EFqwodJ63b 2024-03-25T19:59:11.104 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.EFqwodJ63b is 136KB 2024-03-25T19:59:11.157 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.EFqwodJ63b 2024-03-25T19:59:11.172 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/data/mon.c.tgz 2024-03-25T19:59:11.172 DEBUG:teuthology.orchestra.run.smithi130:> mktemp 2024-03-25T19:59:11.225 INFO:teuthology.orchestra.run.smithi130.stdout:/tmp/tmp.4AKhDlDJVP 2024-03-25T19:59:11.226 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4AKhDlDJVP 2024-03-25T19:59:11.318 DEBUG:teuthology.orchestra.run.smithi130:> sudo chmod 0666 /tmp/tmp.4AKhDlDJVP 2024-03-25T19:59:11.402 DEBUG:teuthology.orchestra.remote:smithi130:/tmp/tmp.4AKhDlDJVP is 158KB 2024-03-25T19:59:11.457 DEBUG:teuthology.orchestra.run.smithi130:> rm -fr /tmp/tmp.4AKhDlDJVP 2024-03-25T19:59:11.471 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/data/mon.b.tgz 2024-03-25T19:59:11.472 DEBUG:teuthology.orchestra.run.smithi163:> mktemp 2024-03-25T19:59:11.489 INFO:teuthology.orchestra.run.smithi163.stdout:/tmp/tmp.GHxrY8QO7v 2024-03-25T19:59:11.489 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.GHxrY8QO7v 2024-03-25T19:59:11.616 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0666 /tmp/tmp.GHxrY8QO7v 2024-03-25T19:59:11.703 DEBUG:teuthology.orchestra.remote:smithi163:/tmp/tmp.GHxrY8QO7v is 167KB 2024-03-25T19:59:11.760 DEBUG:teuthology.orchestra.run.smithi163:> rm -fr /tmp/tmp.GHxrY8QO7v 2024-03-25T19:59:11.775 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T19:59:11.775 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:59:11.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T19:59:11.967 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:59:12.133 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:59:12.207 ERROR:teuthology.util.scanner:Premature end of data in tag valgrindoutput line 3, line 52, column 1 (, line 52) 2024-03-25T19:59:12.373 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T19:59:12.375 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi130/crash 2024-03-25T19:59:12.376 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:59:12.403 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi163/crash 2024-03-25T19:59:12.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T19:59:12.431 INFO:tasks.ceph:Compressing logs... 2024-03-25T19:59:12.431 DEBUG:teuthology.orchestra.run.smithi130:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:59:12.449 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T19:59:12.521 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35100.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.6.log: -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35100.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35100.log.gz 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: 72.0%gzip -- replaced with /var/log/ceph/valgrind/osd.5.log.gz -5 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr: --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T19:59:12.522 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35271.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: 55.8% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35231.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35020.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35231.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35271.log.gz 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35140.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35231.log.gz 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35020.log.gz 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35140.log: 88.6% -- replaced with /var/log/ceph/ceph.audit.log.gzgzip 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T19:59:12.523 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35140.log.gz 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34980.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.34980.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 87.3% -- replaced with /var/log/ceph/ceph-client.admin.34980.log.gz 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35180.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35060.log 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr: 87.0% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/ceph-client.admin.35180.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35180.log.gz/var/log/ceph/ceph-client.admin.35060.log: 2024-03-25T19:59:12.524 INFO:teuthology.orchestra.run.smithi163.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35060.log.gz 2024-03-25T19:59:12.542 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-03-25T19:59:12.542 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41599.log 2024-03-25T19:59:12.542 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40098.log 2024-03-25T19:59:12.542 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40176.log: /var/log/ceph/ceph-client.admin.41599.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T19:59:12.542 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42106.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41599.log.gz 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39708.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40098.log.gz 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.42106.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39708.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41521.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42106.log.gz 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39513.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39708.log.gz 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2024-03-25T19:59:12.543 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41521.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39317.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39122.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41521.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39513.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39317.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39122.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39317.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose 55.8% -- -- replaced with /var/log/ceph/valgrind/mon.c.log.gz /var/log/ceph/valgrind/osd.0.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39122.log.gz 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T19:59:12.544 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/osd.0.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38847.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39161.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39669.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41638.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39161.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38847.log.gz 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39552.log 2024-03-25T19:59:12.545 INFO:teuthology.orchestra.run.smithi130.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39161.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41638.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39669.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40449.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- 87.2% /var/log/ceph/ceph-client.admin.40020.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41638.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39552.log: /var/log/ceph/ceph-client.admin.40449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34987.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40449.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39552.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40215.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.34987.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40332.log 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-25T19:59:12.546 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41989.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.34987.log.gz 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40254.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40215.log.gz 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41989.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40332.log.gz 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39786.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41989.log.gz 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40254.log.gz 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:gzip/var/log/ceph/ceph-client.admin.39786.log: -5 87.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz /var/log/ceph/ceph-client.admin.39747.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T19:59:12.547 INFO:teuthology.orchestra.run.smithi130.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39786.log.gz 2024-03-25T19:59:12.548 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39747.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38966.log 2024-03-25T19:59:12.548 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-25T19:59:12.548 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph.audit.log: 86.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.39747.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38597.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41677.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38597.log: 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.38966.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39005.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.38597.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.39278.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41677.log: /var/log/ceph/ceph-client.admin.39005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41245.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41482.log 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41677.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39005.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41245.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39278.log.gz 2024-03-25T19:59:12.549 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42028.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35465.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41245.log.gz 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.42028.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41482.log.gz 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.35465.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42028.log.gz 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40059.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35465.log.gz 85.8% 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36348.log 2024-03-25T19:59:12.550 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41872.log 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.36348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42067.log 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41872.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.36348.log.gz 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40488.log 2024-03-25T19:59:12.551 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40644.log 2024-03-25T19:59:12.552 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.42067.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41872.log.gz 2024-03-25T19:59:12.552 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41755.log 2024-03-25T19:59:12.552 INFO:teuthology.orchestra.run.smithi130.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42067.log.gz 2024-03-25T19:59:12.552 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41166.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40644.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40488.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41755.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39825.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41166.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40644.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr: 86.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.41755.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39630.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41166.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39044.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7%/var/log/ceph/ceph-client.admin.39630.log: -- replaced with /var/log/ceph/ceph-client.admin.39825.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39200.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39044.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39630.log.gz 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-25T19:59:12.553 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39200.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41206.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39044.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41403.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39200.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39435.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41403.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39981.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41206.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38927.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41403.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39981.log: gzip -5 --verbose 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39435.log.gz 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr: -- /var/log/ceph/ceph-client.admin.39903.log 2024-03-25T19:59:12.554 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38927.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39981.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38927.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41833.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41833.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38808.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40527.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41833.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41127.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38888.log 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38808.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41127.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40527.log.gz 2024-03-25T19:59:12.555 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40605.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.38888.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41127.log.gz 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40605.log: gzip -5 --verbose 87.3% -- replaced with /var/log/ceph/ceph-client.admin.38888.log.gz 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr: -- /var/log/ceph/ceph-client.admin.35293.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41074.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39239.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40605.log.gz 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.35293.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40566.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr: 86.7%/var/log/ceph/ceph-client.admin.39239.log: -- replaced with /var/log/ceph/ceph-client.admin.35293.log.gz 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40566.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40371.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-03-25T19:59:12.556 INFO:teuthology.orchestra.run.smithi130.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40566.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39239.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39083.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40683.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40371.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37400.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39083.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39942.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39083.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.37400.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37400.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39474.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41950.log: 86.1%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39942.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39474.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-03-25T19:59:12.557 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41323.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39474.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41323.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41323.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr: 92.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29135.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41911.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph.tmp-client.admin.29135.log: gzip -5 --verbose 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29135.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr: -- /var/log/ceph/ceph-client.admin.41443.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41284.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41443.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41911.log.gz 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39396.log 2024-03-25T19:59:12.558 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41284.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41443.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41362.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: 86.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41716.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41284.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39396.log: /var/log/ceph/ceph-client.admin.41362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41716.log: 86.9% 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41362.log.gz -- replaced with /var/log/ceph/ceph-client.admin.39396.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40293.log 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41716.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39591.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40293.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39591.log: 87.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39864.log 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40293.log.gz 2024-03-25T19:59:12.559 INFO:teuthology.orchestra.run.smithi130.stderr: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39356.log 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39591.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40137.log: /var/log/ceph/ceph-client.admin.39864.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39864.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41560.log 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.39356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35316.log 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.41560.log: 86.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40410.log 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39356.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: 86.6%/var/log/ceph/ceph-client.admin.35316.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41560.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-client.admin.40410.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35316.log.gz 2024-03-25T19:59:12.560 INFO:teuthology.orchestra.run.smithi130.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40410.log.gz 2024-03-25T19:59:12.588 INFO:teuthology.orchestra.run.smithi163.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T19:59:13.340 INFO:teuthology.orchestra.run.smithi130.stderr:/var/log/ceph/ceph-mon.c.log: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T19:59:13.381 INFO:teuthology.orchestra.run.smithi163.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T19:59:13.428 INFO:teuthology.orchestra.run.smithi163.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T19:59:13.449 INFO:teuthology.orchestra.run.smithi163.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T19:59:13.597 INFO:teuthology.orchestra.run.smithi130.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T19:59:13.614 INFO:teuthology.orchestra.run.smithi130.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T19:59:13.688 INFO:teuthology.orchestra.run.smithi163.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T19:59:13.749 INFO:teuthology.orchestra.run.smithi130.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T19:59:17.397 INFO:teuthology.orchestra.run.smithi130.stderr:gzip: /var/log/ceph/ceph-mon.c.log: file size changed while zipping 2024-03-25T19:59:17.397 INFO:teuthology.orchestra.run.smithi130.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T19:59:17.776 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/ceph-mon.b.log: file size changed while zipping 2024-03-25T19:59:17.776 INFO:teuthology.orchestra.run.smithi163.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T19:59:17.778 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-03-25T19:59:17.778 INFO:teuthology.orchestra.run.smithi163.stderr:real 0m5.285s 2024-03-25T19:59:17.778 INFO:teuthology.orchestra.run.smithi163.stderr:user 0m9.037s 2024-03-25T19:59:17.778 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m0.250s 2024-03-25T19:59:17.821 INFO:teuthology.orchestra.run.smithi130.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T19:59:17.823 INFO:teuthology.orchestra.run.smithi130.stderr: 2024-03-25T19:59:17.823 INFO:teuthology.orchestra.run.smithi130.stderr:real 0m5.363s 2024-03-25T19:59:17.823 INFO:teuthology.orchestra.run.smithi130.stderr:user 0m14.206s 2024-03-25T19:59:17.823 INFO:teuthology.orchestra.run.smithi130.stderr:sys 0m0.479s 2024-03-25T19:59:17.823 INFO:tasks.ceph:Archiving logs... 2024-03-25T19:59:17.824 DEBUG:teuthology.misc:Transferring archived files from smithi130:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi130/log 2024-03-25T19:59:17.824 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:59:20.389 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi163/log 2024-03-25T19:59:20.390 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T19:59:22.000 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:59:22.235 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=55cc19d843f443a4a2a68cda0c5b0ccf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1449, in run_daemon teuthology.stop_daemons_of_type(ctx, type_, cluster_name) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 1173, in stop_daemons_of_type daemon.stop() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 473, in wait check_time() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (51) after waiting for 300 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T19:59:22.241 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T19:59:22.251 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T19:59:22.271 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T19:59:22.271 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:59:22.286 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T19:59:22.369 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T19:59:22.377 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T19:59:22.377 DEBUG:teuthology.orchestra.run.smithi130:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:59:22.380 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T19:59:22.394 INFO:teuthology.orchestra.run.smithi130.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:59:22.396 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-03-25T19:59:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:59:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:=============================================================================== 2024-03-25T19:59:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 55 -558us[ -553us] +/- 68ms 2024-03-25T19:59:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:^* hv02.front.sepia.ceph.com 3 6 377 52 -126us[ -121us] +/- 70ms 2024-03-25T19:59:22.399 INFO:teuthology.orchestra.run.smithi130.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +738us[ +743us] +/- 73ms 2024-03-25T19:59:22.400 INFO:teuthology.orchestra.run.smithi130.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 51 -549us[ -508us] +/- 68ms 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -119us[ -78us] +/- 70ms 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 52 +769us[ +811us] +/- 73ms 2024-03-25T19:59:22.401 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T19:59:22.402 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T19:59:22.411 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T19:59:22.412 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T19:59:22.419 DEBUG:teuthology.orchestra.run.smithi130:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:59:22.512 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T19:59:22.583 DEBUG:teuthology.orchestra.run.smithi130:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:59:22.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:22.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T19:59:22.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:22.647 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T19:59:22.656 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T19:59:22.665 INFO:teuthology.task.internal:Duration was 5683.085364 seconds 2024-03-25T19:59:22.665 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T19:59:22.674 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T19:59:22.674 DEBUG:teuthology.orchestra.run.smithi130:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:59:22.677 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T19:59:22.721 INFO:teuthology.orchestra.run.smithi130.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:59:22.732 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T19:59:23.070 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T19:59:23.070 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi130.front.sepia.ceph.com 2024-03-25T19:59:23.070 DEBUG:teuthology.orchestra.run.smithi130:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:59:23.091 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-03-25T19:59:23.091 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T19:59:23.113 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T19:59:23.113 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:59:23.134 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T19:59:23.182 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T19:59:23.182 DEBUG:teuthology.orchestra.run.smithi130:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:59:23.204 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T19:59:23.325 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T19:59:23.334 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T19:59:23.334 DEBUG:teuthology.orchestra.run.smithi130:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:59:23.362 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T19:59:23.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T19:59:23.399 DEBUG:teuthology.orchestra.run.smithi130:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:23.405 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:23.428 INFO:teuthology.orchestra.run.smithi130.stdout:kernel.core_pattern = core 2024-03-25T19:59:23.455 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-03-25T19:59:23.490 DEBUG:teuthology.orchestra.run.smithi130:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:23.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:23.505 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T19:59:23.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T19:59:23.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T19:59:23.558 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T19:59:23.559 DEBUG:teuthology.misc:Transferring archived files from smithi130:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi130 2024-03-25T19:59:23.559 DEBUG:teuthology.orchestra.run.smithi130:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:59:23.595 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322/remote/smithi163 2024-03-25T19:59:23.595 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T19:59:23.635 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T19:59:23.635 DEBUG:teuthology.orchestra.run.smithi130:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:59:23.638 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T19:59:23.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T19:59:23.696 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T19:59:23.696 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T19:59:23.706 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T19:59:23.707 DEBUG:teuthology.orchestra.run.smithi130:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:59:23.709 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T19:59:23.722 INFO:teuthology.orchestra.run.smithi130.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:59 /home/ubuntu/cephtest 2024-03-25T19:59:23.740 INFO:teuthology.orchestra.run.smithi163.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 19:59 /home/ubuntu/cephtest 2024-03-25T19:59:23.741 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T19:59:23.752 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T19:59:23.776 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T19:59:23.800 DEBUG:teuthology.nuke:shortname: smithi130 2024-03-25T19:59:23.800 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:59:23.823 DEBUG:teuthology.nuke:shortname: smithi163 2024-03-25T19:59:23.823 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T19:59:23.826 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi130.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:15:20.298507', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBC151G++LGPgESxw1yLJIi18NohTBabOkdI3F3RowAdjaQ+fN969rAq2YN8RW/M5KRLIbbjm23W8VU5H6WnoUZA='} 2024-03-25T19:59:23.842 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621322', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 18:15:20.299504', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBoqCjQcNIhRg3dFSzdt0NrGct9OOSowWbTVZL//H+1UrhCpY21F4j7tYptMsoDnwA2eSnEepdYgb6Ac2LGNpcQ='} 2024-03-25T19:59:23.879 INFO:teuthology.orchestra.console.smithi130:Power off 2024-03-25T19:59:23.879 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:59:23.895 INFO:teuthology.orchestra.console.smithi163:Power off 2024-03-25T19:59:23.895 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T19:59:23.907 DEBUG:teuthology.orchestra.console.smithi130:power off output: Chassis Power Control: Down/Off 2024-03-25T19:59:23.908 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:23.921 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-03-25T19:59:23.921 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:23.935 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-25T19:59:23.946 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-25T19:59:27.937 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:27.947 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:28.048 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-25T19:59:28.057 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-25T19:59:32.050 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:32.059 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:32.161 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is on 2024-03-25T19:59:32.169 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-03-25T19:59:36.166 DEBUG:teuthology.orchestra.console.smithi130:pexpect command: ipmitool -H smithi130.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:36.176 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T19:59:36.278 DEBUG:teuthology.orchestra.console.smithi130:check power output: Chassis Power is off 2024-03-25T19:59:36.287 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-03-25T19:59:36.380 INFO:teuthology.orchestra.console.smithi130:Power off completed 2024-03-25T19:59:36.389 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-03-25T19:59:36.494 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 5683.085364103317 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=55cc19d843f443a4a2a68cda0c5b0ccf status: fail success: false 2024-03-25T19:59:36.494 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T19:59:36.576 INFO:teuthology.run:FAIL